acceptodds
Under review as a conference paper at ICLR 2027

Do Streaming Video Models Remember The Past? Verifying and Evolving Memory System for Videos

Abstract

Memory management is a fundamental capability for long-horizon video understanding, where models must store and retrieve useful information from a growing stream of observations. Yet, it remains unclear whether existing benchmarks actually require the memory, as even a simple memory-free system can outperform dedicated memory-augmented models on standard streaming-video benchmarks, raising questions about what these evaluations truly capture. We introduce MemProof, a benchmark that verifies memory dependence by ensuring each question is unsolvable from language or recent observations alone, but becomes solvable once the necessary past evidence is revealed. With memory dependence explicitly verified, we next ask whether the memory-management strategy itself can be improved without updating any model parameters. We introduce EvoHarness, which keeps all neural components frozen and evolves modular programs for organizing, retrieving, and presenting memory through teacher-guided failure diagnosis and paired validation. Experiments show that strong performance on conventional streaming benchmarks can be achieved even without memory, whereas MemProof requires models to actually use the past: under the same backbone, evolution improves the base harness by +8.0 points, and the memory organization and packing mechanisms of EvoHarness transfer to other benchmarks. The code/data will be made available online.

Primary area:
datasets and benchmarks
Keywords:
Long Video Understanding, Streaming Video Understanding, Video Understanding Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_r36NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-term memory in streaming video from two perspectives. It introduces MemProof to explicitly verify whether questions truly depend on historical visual evidence, and EvoHarness to iteratively improve external memory organization, retrieval, and presentation while keeping neural models frozen. The experiments show that strong performance on conventional streaming benchmarks does not necessarily imply effective memory use, while EvoHarness achieves clear gains on MemProof.

Critical strengths
  1. The motivation is compelling and timely. The paper clearly distinguishes between questions that merely refer to the past and those that genuinely require historical evidence, exposing an important evaluation gap in current streaming-video benchmarks.

  2. The design of EvoHarness is also pretty interesting. Instead of updating model parameters, it improves the external memory pipeline through failure diagnosis and validated program-level changes, making the optimization process relatively interpretable.

Critical weaknesses
  1. The practical efficiency of EvoHarness in a true streaming setting needs to be evaluated. EvoHarness involves multiple stages, while the paper reports the number of retrieved frames, it does not provide latency, throughput, memory usage, or a runtime breakdown, making its practical suitability for real-time streaming difficult to assess.
Questions for authors
  1. After all benchmark construction, filtering, and data splitting steps, was the finalized MemProof benchmark independently re-run through the full blind/recent/evidence and multi-event verification pipeline?

  2. Could you report the practical inference efficiency of EvoHarness, including latency, throughput, GPU memory, and ideally a per-module runtime breakdown?

  3. What is the main deployment-time computational bottleneck in EvoHarness? Have you considered incorporating latency or computational cost into the harness-evolution objective so that the system jointly optimizes accuracy and efficiency?

4 comments
AuthorsOpenReview ↗

Author response to (W1,Q2) Practical inference efficiency in the streaming setting.

(W1,Q2) Practical inference efficiency in the streaming setting. We thank the reviewer for this suggestion. In short, H6H_6 answers a question in about 1.5 s on a single GPU, about 0.5 s slower than uniform frame sampling. Accessing the memory itself takes less than 0.1 s; most of the time goes to the two LLM calls of the harness, the query planner (planner.py) and the final answer (asnwer.py), and to preparing the selected frames for the model (packer.py: CPU job).

Setup. We ran the pipeline of Fig. 2 of the main paper on one H100 GPU, with the same frozen models: Qwen3.5-9B as the planner and the agent f(⋅)f(\cdot), and Qwen3-Embedding-4B as the text embedder. As timing on the full 4,396-question test split would take too long, we used 216 test questions sampled across the six dimensions. We compare H0H_0, H3H_3, and H6H_6 with memory-free baseline: Uniform 28+4 (Tab. 5 of the supplement).

Latency and throughput. We report two measures in Tab. R3. Latency is the time to answer one question when it is the only request; we give the median (p50) and the 95th percentile (p95). Throughput is the number of questions answered per second when 8 or 16 requests arrive at the same time. H6H_6 answers in 1.53 s at the median and 2.48 s at the 95th percentile, compared with 1.06 s and 1.99 s for Uniform 28+4. With 16 simultaneous requests, H6H_6 answers 2.5 questions per second, against 3.0 for Uniform 28+4.

Tab. R3. Latency and throughput on one H100 GPU (216 test questions, Qwen3.5-9B). Latency: time to answer one question when it is the only request. Throughput: questions answered per second with 8 or 16 simultaneous requests.

SystemLatency (s): p50Latency (s): p95Throughput (q/s): 8 requestsThroughput (q/s): 16 requests
H0H_01.282.463.154.10
H3H_31.542.572.493.00
H6H_61.532.482.382.51
Uniform 28+41.061.993.062.98

Runtime per module (Tab. R4). We timed each program of Fig. 2 of the main paper. For H6H_6, the LLM call of answer.py takes 0.69 s, the LLM call of planner.py 0.41 s, and preparing the selected frames for the model 0.41 s. Frame preparation in packer.py runs on the CPU, not the GPU: each selected frame is loaded from disk, resized, and encoded into the request sent to the model. Its cost therefore grows with the number of frames delivered to the model. retriever.py, which embeds the queries and matches them against the MacroEvents, takes 0.05 s, and reading the memory built by builder.py takes 2 ms.

Tab. R4. Runtime per module (mean seconds per question, one request at a time). Programs follow Fig. 2 of the main paper. Frame preparation runs on a CPU that was shared with other jobs during the measurement. Uniform 28+4 does not use these programs, so only its total time is shown.

ModuleH0H_0H3H_3H6H_6Uniform 28+4
builder.py: reading the memory0.0020.0030.002--
planner.py: LLM call0.4120.4160.408--
retriever.py: query embedding and matching0.0590.0590.054--
packer.py: preparing the selected frames0.4440.4330.405--
answer.py: LLM call(s)0.4850.6710.692--
Other0.0030.0040.009--
Total1.4051.5851.5701.178

GPU memory. The engine serving the planner and f(⋅)f(\cdot) holds 17.7 GiB of weights; with the memory it reserves for serving requests, its peak use is 65.7 GiB of the 95 GiB card. The embedder adds 7.6 GiB of weights. The memory itself is a small text structure stored on disk, 47 KB per video on average.

Correction of the supplement. Sec. 4.7 of the supplement reports 5.6 s per question for H3H_3 on OVO-Bench. These are average times per question in 'large batch' runs, in which about 32 requests shared 8 engines and waited for each other; they are not the latency of one question. Measured alone, H3H_3 takes 1.54 s. We will revise it accordingly.

AuthorsOpenReview ↗

Author response to (Q1) Re-certification of the finalized benchmark.

(Q1) Re-certification of the finalized benchmark. We thank the reviewer for the suggestion. In short, the final benchmark had not been re-certified as a whole before submission, so we have now re-run the full verification on the final test questions.

Setup. Recall we verify each question truly needs the past in three stages: blind, recent, and evidence (main Sec. 2.2). A question "passes" only if the verifiers fail in the first two stages but succeed in the third. Multi-event questions must also pass the multi-event tests (main Sec. 2.3). These checks, however, were run on candidate questions, before the final benchmark was assembled. During assembly, some questions were reworded and some had their options reshuffled, so a question that passed as a candidate is not guaranteed to pass in its final form. As the reviewer points out, re-certifying the final benchmark is therefore the right way to confirm that its questions still need the past. From 4,396 test questions, we excluded the 750 implicit-verification questions, (as they were not built with the three-stage verification) and performed re-certification on the remaining 3,646 questions. We took the final test questions exactly as released and passed each of them through the same checks again, as if it were a new candidate: the same two verifiers, the same prompts, and the same pass rules.

Results. Of the 3,646 re-certified test questions, 3,296 (90.4%) passed and 350 failed. (The 750 implicit-verification questions were not re-certified, as explained above, and are not counted in either number.) The failures have three main causes: (i) Some questions were reworded after their original check, and the new wording can change how the verifiers respond. (ii) The verifiers do not give exactly the same answers on every run, so questions close to a pass threshold can change outcome. (iii) Some multi-event questions are labeled as needing multiple past events (Wide), but in the re-check they could be answered with fewer of these events. They still cannot be answered without the past, so they remain memory questions; they only fall to a lower level (Narrow instead of Wide), which our rule counts as a failure. We will include this analysis in our final manuscript.

AuthorsOpenReview ↗

Author response to (Q3) Deployment bottleneck and cost-aware evolution.

(Q3) Deployment bottleneck and cost-aware evolution. The main bottleneck is the LLM calls, not the memory. Of the 1.57 s of an H6H_6 question (Tab. R4), the answer call (answer.py) takes 0.69 s, the planner call (planner.py) 0.41 s, and frame preparation on the CPU (packer.py) 0.41 s, while memory access takes less than 0.06 s. The planner is the only extra cost of the memory system, matching the 0.4 s gap to Uniform 28+4. Cost can thus be cut by lighter LLM use (e.g., a smaller planner), caching the prepared frames, or more GPUs.

Our current evolution does not optimize latency or computational cost; it focuses only on improving accuracy, since the acceptance rule keeps a proposal based on its accuracy gain alone (main Sec. 3.2). The only cost constraint is the fixed 32-frame budget (supp Sec. 3.1). Under this accuracy-only objective, frames per question happened to decrease (18.9→17.018.9 \rightarrow 17.0), but LLM calls per question increased (2.00→2.762.00 \rightarrow 2.76; supp Tab. 8). Incorporating cost is straightforward, since each receipt already records frames, calls, and tokens; for example, the acceptance rule could reject proposals that exceed a limit on LLM calls or latency, in the same way as the frame budget. Our current manuscript leaves out these experiments but will include it as future work.

Thanks for the rebuttal with detailed answers. Most of my concerns are addressed, and I have no further questions.

Reviewer_guabOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that high scores on existing streaming-video benchmarks do not necessarily show that a model remembers the past. It introduces MemProof, which tests whether questions require historical evidence, and EvoHarness, which improves memory organization, retrieval, and presentation while keeping the neural models frozen. EvoHarness raises the MemProof score from 58.4 to 66.4; results on other benchmarks and an alternative answering model show some transfer, though the gains are not consistent across all settings.

Critical strengths
  1. The paper distinguishes questions that refer to past events from questions that require past evidence. The strong results of a recent-four-frame system on the reported subsets of existing streaming benchmarks provide a concrete motivation for examining what those benchmarks measure (main paper, pp. 1–2).

  2. MemProof tests answers under blind, recent-only, and evidence-revealed conditions, and separates recent perception, narrower and wider historical evidence, difficult-with-evidence questions, and premise verification. Its 12,207 questions over 5,437 videos use video-disjoint training and test splits (main paper, pp. 3–4; supplement, p. 3). This is more informative than inferring memory demand from a question’s wording alone.

  3. EvoHarness separates memory construction, planning, retrieval, evidence packing, and answer presentation into external programs. Neural components remain frozen during evolution, while inference is constrained to observations available before the query and a fixed visual-input budget (main paper, pp. 5–7; supplement, pp. 6–8).

Critical weaknesses
  1. The claim that memory dependence is “certified” is stronger than the filtering protocol supports. Template-route questions can pass with blind accuracy as high as 0.5. For generative-route questions, a blind or recent candidate is rejected only when both verifiers select the gold answer with sufficient confidence; an item that one verifier answers correctly without historical evidence can therefore survive. The protocol establishes dependence relative to the selected verifiers and thresholds, rather than strict unanswerability without history (main paper, p. 4, Eq. 2; supplement, pp. 3, 15).

  2. The benchmark and harness share a caption substrate that materially affects results. Questions are generated from Qwen3-VL captions, and the harness uses captions from the same source as memory. H6 scores 62.5 on Wide with frames and 61.9 when all frames are removed from the answering model’s input. Rebuilding memory with a Gemma captioner lowers H6’s overall score from 66.4 to 56.9 and markedly reduces the later verification gains (supplement, pp. 10–11, Table 6). Historical captions are a valid form of memory, and changing captioners also changes caption quality. The result nevertheless limits claims about robustness to independently represented visual evidence.

  3. Parts of the aggregate evaluation are vulnerable to skew or small-sample weighting. The gold answer is the first option for 95.7% of Recent multiple-choice questions. The authors’ option-derangement test on 629 validation questions is reassuring for the tested harnesses, but it does not establish robustness for every evaluated system on the test set. Separately, just 46 gradable Ceiling questions receive one-sixth of the All score; their contribution alone exceeds H0’s reported two-point aggregate advantage over the 68-frame uniform control (supplement, pp. 5, 13–14). All is best interpreted as a balanced diagnostic objective, not a conventional per-question measure of historical-memory accuracy.

  4. The contribution of the evolution procedure, and the scope of its gains, remain incompletely isolated. The H0→H6 test gain establishes that the resulting program changes help, but there is no equal-budget comparison with manual tuning, generic program optimization, or proposals made without ground-truth-evidence diagnosis (supplement, p. 15). The H3→H6 stage gains five points on MemProof but only −1.1 to +1.2 points on the external benchmarks reported in the supplement. On OVO-Bench proactive responding, both H3 and H6 trail a same-backbone uniform control; on OVO Backward, H3 takes 5.6 seconds per question versus 1.1 seconds for the recent-four-frame control (supplement, pp. 12, 15). The evidence supports useful early memory-organization mechanisms more strongly than a general improvement across all streaming tasks or resource budgets.

Questions for authors
  1. What exactly does each certification route establish? For generative-route questions, is an item retained when one verifier answers correctly without history but the other does not? If so, how is the claim that the item is “unanswerable” without history intended to be interpreted?

  2. How answerable is the question after evidence is removed? For retained multi-event questions, what are the absolute scores—not only relative score drops—under singleton, leave-one-out, alternative-evidence, and bipartition inputs? This determines whether evidence width reflects strict necessity or a measurable contribution from each event.

  3. How robust is the aggregate metric to its known design choices? What chance-correction formula is used for Recent multiple-choice questions with 95.7% of gold answers in the first position, and how do the test results change under balanced option positions and with the 46 gradable Ceiling questions reported separately? These checks could change the interpretation of All.

  4. Do the gains persist when question wording and memory captions are independent? Under a controlled captioner or question-generation change that preserves the underlying video facts, how much of H0→H6—and especially H3→H6—remains? The reported caption-only and captioner-swap results make this decisive for the claimed generality of the verification mechanisms.

5 comments
AuthorsOpenReview ↗

Author response to (W1,Q1) What each certification route establishes.

(W1,Q1) What each certification route establishes. We appreciate the reviewer pointing this out. In short, memory dependence in MemProof is established by the combination of the three verification stages:

(i) The blind threshold is deliberately loose (0.5), so that a question is not discarded just because a verifier happened to guess its answer correctly (lucky guess). A question guessable from language alone would still be caught in the next, recent stage, which sees the same question under a stricter threshold (0.25). In practice, the questions that pass blind stage are almost never answered without video: on the template route, the verifiers answer them correctly only 1–3% of the time when given the question alone.

(ii) On the generative route, a question is indeed kept even when one verifier answers it correctly without video. Each verifier answers only once on this route, so a single correct answer may just be luck, and we drop a question only when both verifiers get it right. These questions may also be partly answerable from the question alone (42% and 31% in the blind stage), but this does not carry over to the recent stage, which reflects the actual streaming setup. Once the recent frames are added, which show a different part of the video, accuracy falls to 24% and 7%, no better than random guessing (25% for four options). In the final evidence stage, it rises to nearly 100% (100% and 99.7%).

(iii) To verify that this result does not depend on the two verifiers, we additionally tested a third model (Qwen3.5-9B). As shown in Tab. R5, it also fails on the memory questions when it sees only the question or the recent frames, and succeeds only when it is also shown the past. Its small score on Wide from the question alone (21.0) disappears once the recent frames are shown.

Tab. R5. The three verification stages applied to a model not used for certification (Qwen3.5-9B, our answering model). Each row indicates the model input: the question alone, the question with the recent frames RtR_t (last 4 frames), or the question with the annotated past evidence Et⋆E^{\star}_t. In this table, scores are computed as (accuracy − 25%) / 75% × 100 for four-option questions, so that random guessing scores 0 and a fully correct answer scores 100; other formats use their own random-guessing accuracy in place of 25% (e.g., 50% for yes/no). For example, on a four-option question, an accuracy of 25% (random guessing) gives a score of 0, 62.5% gives 50, and 100% gives 100. Values slightly below 0 are shown as 0.

InputRecentNarrowWide
Question only (blind)54.20.021.0
+ recent frames (recent)96.75.30.0
+ past evidence (evidence)97.859.178.2

Each route thus establishes the same property in a slightly different way. On the template route, a question is rarely answered from qtq_t alone or from RtR_t; on the generative route, it may be guessed by one verifier from qtq_t alone, but not from RtR_t. On both routes, it becomes answerable once Et⋆E^{\star}_t is shown. "Unanswerable without history" therefore means that the verifiers cannot answer the question from what is available at query time, namely qtq_t together with RtR_t, but can once Et⋆E^{\star}_t is shown. In this sense, we use the term "certified" to mean that a question has passed our verification procedure, not that it is proven impossible to answer without history. We will state this definition and the rule of each route explicitly in Sec. 2.2.

AuthorsOpenReview ↗

Author response to (W2,Q4) Robustness to question wording and memory captions.

(W2,Q4) Robustness to question and caption wording. We thank the reviewer for this point. In short, the harness does not score by matching words between (a) the questions (generated from Qwen3-VL-8B), and (b) the captions stored in its memory (come from the same Qwen3-VL-8B). To show this, we performed two experiments that reword the questions and the captions, respectively, while keeping their facts unchanged, as described below.

Experimental setup. Recall that the captioner swap experiment (supp Sec. 4.2 and Tab. 6) changes only (b), by replacing captioner, Qwen3-VL-8B, with Gemma-4-26B. Since the two models differ in what they describe and how accurately, this swap changes the content and quality of the captions together with their wording. To isolate wording, we therefore ran two paraphrase experiments, which keep the facts fixed and change only the wording, one for each side:

  • (A) Question paraphrase: only the questions are reworded; the memory captions are unchanged.
  • (B) Caption paraphrase: only the memory captions are reworded; the questions are unchanged.

In (A), an LLM (Gemma-4-26B) rewrote the question and options, keeping the meaning, the answer, and the option order, and a second LLM (Qwen3.6-35B-A3B) checked that each rewrite kept the same meaning; 94.5% of the rewrites were accepted. In (B), the same two LLMs rewrote (Gemma-4-26B) and checked (Qwen3.6-35B-A3B) every caption, and about 95% of the captions were reworded. We then rebuilt the memory from these captions with the original procedure. In both (A) and (B), we ran H0, H3, and H6 with all neural models frozen, and compared each harness with its result on the original text in Tab. R6. Each comparison is restricted to the questions affected by the rewording, so that the original and reworded runs are evaluated on exactly the same questions: in (A), the questions whose rewrite was accepted by the checker; in (B), the questions whose memory was affected by the caption rewording.

Tab. R6. Gain of each evolution stage with reworded questions or captions (MemProof test split).

Each entry is the average per-question score increase from one harness to the next, computed on the subset affected by each experiment: (A) the questions with an accepted rewrite; (B) the questions whose memory was affected by the caption rewording. Original: the same subset before rewording. Reworded questions: questions and options paraphrased, memory unchanged. Reworded captions: memory rebuilt from paraphrased captions, questions unchanged. Because (A) and (B) use different subsets, their “Original” columns differ from each other and from the full-test per-question gains in Tab. R8.

Stage(A) Original(A) Reworded questions(B) Original(B) Reworded captions
H0→H3H_0 \rightarrow H_3+4.4+3.9+4.8+4.2
H3→H6H_3 \rightarrow H_6+7.4+3.8+6.8+6.7
H0→H6H_0 \rightarrow H_6+11.8+7.7+11.6+10.9

(A) Rewording the questions leaves the early stage unchanged (H0→H3H_0 \rightarrow H_3: +3.9+3.9 vs. +4.4+4.4 points), but halves the later stage (H3→H6H_3 \rightarrow H_6: +3.8+3.8 vs. +7.4+7.4). Investigating the cause, we found that several H6H_6 mechanisms (e.g., boundary captions and the claim-specific retrieval (main Tab. 2)), are switched on only when the question contains a phrase of our templates, such as "best sums up". Rewording removes these phrases, so the mechanisms almost never switch on. This dependence arises because the evolution saw only our template phrases, and it can be reduced by rewording each training question in several ways during evolution, so that the mechanisms learn to recognize the question type rather than a fixed phrase.

(B) Rewording the memory captions leaves every stage unchanged (H0→H3H_0 \rightarrow H_3: +4.2+4.2 vs. +4.8+4.8; H3→H6H_3 \rightarrow H_6: +6.7+6.7 vs. +6.8+6.8). The harness therefore relies on the facts in the captions, not on their exact wording. This is consistent with the caption-only result noted by the reviewer: H6H_6 draws most long-term evidence from the captions rather than from the frames. Storing the past as text is the intended design of our memory, and we will clarify in the paper that our harness remembers the past mainly through text, not by reasoning over the past video frames.

Together, these experiments clarify the role of wording. Experiment (A) shows that question wording did shape which mechanisms were proposed during evolution, since some mechanisms rely on our templates phrases. Experiment (B) shows that caption rewording leaves every gain unchanged. Taken together, both (A) and (B) show that the gains themselves do not come from wording shared between the questions and the captions. The larger drop under the captioner swap thus most likely reflects a change in what the captions describe, rather than in how they are worded.

AuthorsOpenReview ↗

Author response to (W3,Q3) Robustness of the aggregate metric.

(W3,Q3) Robustness of the aggregate metric. We thank the reviewer for pointing out two limitations of the aggregate score 'All' (main Tab. 3): (i) the skewed answer position in 'Recent', and (ii) the large weight of the small 'Ceiling' dimension. We tested both (i) by balancing the option order and (ii) by recomputing 'All' without 'Ceiling', and found that neither changes our main conclusions. In these experiments, each question is scored as (accuracy − random) / (100% − random) × 100, where random is the accuracy of random guessing for its format (e.g., 25% for four options and 50% for yes/no), so that random guessing scores 0 and a fully correct answer scores 100, making scores comparable across question formats. The experiment details are described below.

(i) Balancing option order. In 95.7% of the 'Recent' questions, the GT is the first option ("A"), so a model that tends to choose "A" could score high without understanding the video. To test this, we shuffled the options of the 1,093 'Recent' so that GT option appears in each of the four positions equally often, and re-ran the systems on the test split (Tab. R7). A policy that always answers "A" falls from 95.0 to 0.1, confirming that the shuffle removes the skew. In contrast, H6H_6 is unchanged (97.3→97.297.3 \rightarrow 97.2), and H0H_0 even improves. The skew therefore did not inflate our harnesses, and its effect on 'All' is at most 0.7 points. We will add this to the supplement and release the balanced 'Recent'.

Tab. R7. Recent scores before and after balancing the GT position.

SystemOriginalBalancedChange
Always "A"95.00.1−94.9
H0H_090.494.4+4.0
H6H_697.397.2−0.1

(ii) Recomputing 'All' without 'Ceiling'. To test how much the final score depends on 'Ceiling', we recomputed every comparison in two other ways (Tab. R8): All without 'Ceiling', and the per-question score, which averages over all 4,352 graded questions so that every question counts equally. The evolution gains do not depend on 'Ceiling'. The overall gain even grows without it (H0→H6H_0 \rightarrow H_6: +8.0+8.0 in 'All', +9.4+9.4 without 'Ceiling', +9.1+9.1 per question), and so does the later stage (H3→H6H_3 \rightarrow H_6: +5.0+5.0 to +6.5+6.5 and +6.7+6.7), while the early stage stays similar (H0→H3H_0 \rightarrow H_3: +2.4+2.4 to +3.0+3.0). H6H_6 also exceeds the memory-free Uniform 64+4 (supp Tab. 5). As the reviewer noted, however, the 2-point advantage of H0H_0 over Uniform 64+4 comes from 'Ceiling' and disappears without it. We therefore do not claim that H0H_0 is better than Uniform 64+4, only that it matches it while using 72% fewer frames, as the supplement already states.

Tab. R8. Score differences under three ways of aggregating. All: the six-item objective of the main paper. w/o Ceiling: average of the other five items. Per question: average over all 4,352 graded questions. Uniform 64+4: memory-free control that shows 64 uniformly sampled past frames and the 4 recent frames (supp Tab. 5).

ComparisonAllw/o CeilingPer question
H0→H3H_0 \rightarrow H_3+3.0+2.9+2.4
H3→H6H_3 \rightarrow H_6+5.0+6.5+6.7
H0→H6H_0 \rightarrow H_6+8.0+9.4+9.1
Uniform 64+4 →H6\rightarrow H_6+10.0+9.1+7.7
Uniform 64+4 →H0\rightarrow H_0+2.0−0.3−1.3
Reviewer_s1JmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether streaming video benchmarks actually require long-term memory and whether memory management can be improved without updating model parameters. It introduces MemProof, a benchmark that verifies memory dependence using blind, recent-only, evidence-revealed, and multi-event necessity tests, and EvoHarness, a modular framework that iteratively improves memory organization, retrieval, and evidence presentation through teacher-guided failure diagnosis and paired validation. On the MemProof test set, the final evolved harness improves the chance-corrected six-item objective from 58.4 to 66.4 relative to the base harness. Evaluations on existing streaming and long-video benchmarks further indicate that some of the evolved memory-organization and evidence-packing mechanisms transfer beyond MemProof.

Critical strengths
  1. MemProof addresses an important evaluation gap in streaming video understanding. Its blind, recent-only, evidence-revealed, and multi-event necessity tests distinguish questions that actually require historical evidence. This enables a more reliable evaluation of long-term video memory.
  2. EvoHarness provides a modular and interpretable approach to improving memory use without updating neural parameters. By diagnosing failures in memory organization, retrieval, and presentation, and validating proposed program-level changes, the framework improves the MemProof six-item objective from 58.4 to 66.4. The transfer of some organization and packing mechanisms to external benchmarks further supports the practical relevance of the approach.
Critical weaknesses
  1. The contribution of the teacher LLMs is not sufficiently isolated. The authors filter, merge, and manually implement the LLM-generated proposals before retaining only validated changes, but do not systematically report how many original proposals were rejected, implemented, individually validated, or preserved after combination. Consequently, it is unclear whether the gains reflect consistently useful LLM diagnoses and proposals or primarily effective human curation and validation-based selection. Reporting this proposal funnel and the validation gains of individual proposals would better establish the teacher models’ contribution.
  2. Cross-backbone evidence is limited. The study evaluates only two Qwen-family models, and not all evolution stages transfer consistently. Results on distinct VLM families are needed to establish backbone-independent generalization.
Questions for authors
  1. Could the authors provide a complete proposal funnel for each evolution round, including the numbers of teacher-generated proposals that were filtered or merged, implemented, individually validated, and retained after combination, together with their proposal-level validation gains? This clarification would help determine how much of the improvement is attributable to the teacher-generated diagnoses and proposals rather than human curation and validation-based selection.
  2. Could the authors report the performance of EvoHarness with answering VLMs from non-Qwen model families to better assess its cross-backbone generalization?
2 comments
AuthorsOpenReview ↗

Author response to (W1,Q1) Complete proposal funnel for each evolution round.

(W1,Q1) Complete proposal funnel for each evolution round. We thank the reviewer for this question. Tab. R1 reports the complete funnel. It shows that the teachers, not the authors, wrote the mechanisms, and that validation only decided which of them to keep. Over the six evolution steps, the two teachers wrote 125 proposals; 53 remained after cross-review and were validated (68 validation runs in total, including 15 re-evaluations of combined mechanisms before promotion), and 15 were promoted to form H1,…,H6H_1, \dots, H_6. We explain the funnel below, and we will add the paired comparison (+/−) of every validated proposal to the supplement.

How the evolution works. We first recall the terms in Sec. 3.2 of the main paper. A mechanism is a program-level change to the harness, such as the event counts added to builder.py (Tab. 2). At each evolution step, the two teacher LLMs read the failure diagnosis of HkH_k on Ddev\mathcal{D}_{\text{dev}}, and each writes proposals, i.e., candidate mechanisms. In cross-review, the two sets of proposals are compared: redundant proposals are merged, and unsupported ones are discarded. Each remaining proposal is implemented on top of HkH_k and validated: it is evaluated against HkH_k on Dval\mathcal{D}_{\text{val}}, counting the examples it corrects and breaks (the +/−+/- of Tab. 2). Proposals that pass are promoted, i.e., added to HkH_k to form Hk+1H_{k+1}.

The gains come from the teachers. (i) The authors did not write or choose any mechanism. We defined the rules of the evolution (data partitions, the acceptance rule, and the budget constraints) and supervised it. Following these rules, cross-review, implementation, and the promotion decisions were carried out by LLMs, correcting Sec. 3.3 of the supplement, which states that the authors merged/implemented the proposals. The gains therefore do not come from human curation. (ii) Most promoted mechanisms were proposed by the teachers. H6H_6 contains 15 mechanisms in total (Sec. 4.6 of the supplement). Tab. 2 and Tab. 4 of the supplement list the main ones (12 proposals), and we will list all 15 in the final manuscript. (iii) The acceptance rule only selects; it does not create mechanisms. It can keep or reject a proposal, but it cannot write a new one, and it rejects most of them: of the 125 proposals, only 15 mechanisms were promoted. After H6H_6, two further evolution iterations promoted none of their proposals (Sec. 4.5 of the supplement), which is why the evolution stopped at H6H_6. In short, the teachers write the mechanisms, and the acceptance rule decides which ones to keep.

Tab. R1. Proposal funnel of each evolution step. Proposed: proposals of teacher A (Claude Fable 5) / teacher B (GPT-5.6 Sol), each written without seeing the other's. After cross-review: proposals kept after cross-review, as written or after modification. Promoted: mechanisms added to form the harness of that row (after validation step). +/−+/-: examples corrected/broken by the promoted mechanisms relative to the previous harness, as in Tab. 2.

HarnessProposed (A / B)After cross-reviewPromoted++ / −-
H1H_112 / 1210687 / 53
H2H_212 / 122233 / 15
H3H_36 / 124123 / 9
H4H_44 / 128281 / 48
H5H_511 / 1214114 / 6
H6H_68 / 1215328 / 18
Total53 / 725315
AuthorsOpenReview ↗

Author response to (W2,Q2) Transfer to non-Qwen answering backbones.

(W2,Q2) Transfer to non-Qwen answering backbones. We thank the reviewer for this suggestion. We evaluated the fixed H0H_0, H3H_3, and H6H_6 harnesses with two answering VLMs from different model families: Gemma-4-E4B-it (Google) and Ministral-3-8B-Instruct (Mistral); only the answering model f(⋅)f(\cdot) is replaced while others (e.g., planner, embedder) are unchanged and the harness is not re-evolved for the new models. Tab. R2 summarizes the results.

Tab. R2. Fixed harnesses with non-Qwen answering models on the MemProof test split. All: the six-item objective. Uniform: 4 recent + 28 uniform frames. Δ\Delta: increase of All between two harnesses.

Answering modelFamilyAll: UniformAll: H0H_0All: H3H_3All: H6H_6Δ\Delta: H0→H3H_0 \rightarrow H_3Δ\Delta: H3→H6H_3 \rightarrow H_6Δ\Delta: H0→H6H_0 \rightarrow H_6
Gemma-4-E4B-itGoogle36.848.651.957.4+3.3+5.5+8.8
Ministral-3-8B-InstructMistral32.843.045.949.1+2.9+3.2+6.1
Qwen3.5-9BQwen54.258.461.466.4+3.0+5.0+8.0

Results. (i) The evolved harness improves both non-Qwen models. From H0H_0 to H6H_6, All increases by +8.8+8.8 points for Gemma and +6.1+6.1 for Ministral, comparable to +8.0+8.0 for the Qwen3.5-9B reference. (ii) Both evolution stages transfer. Here, All increases at both stages on both new models: by +3.3+3.3 and +2.9+2.9 from H0H_0 to H3H_3, and by +5.5+5.5 and +3.2+3.2 from H3H_3 to H6H_6. (iii) The gains concentrate on memory-heavy questions. On Wide, H6H_6 raises Gemma from 35.0 to 60.6 and Ministral from 41.0 to 58.8. This is consistent with the gains coming from memory-side mechanisms rather than from tuning to one answering model. (iv) On every model, H6H_6 clearly exceeds the Uniform, by +20.6+20.6 (Gemma), +16.3+16.3 (Ministral), and +12.2+12.2 (Qwen3.5-9B) points. We will include these results in the final manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank all reviewers for their thoughtful and constructive reviews. We especially appreciate that the reviewers recognized the main strengths of our work:

  • An important and timely problem. Our work distinguishes questions that merely refer to the past from those that genuinely require it, exposing an evaluation gap in current streaming-video benchmarks (s1Jm, guab, r36N).
  • Verified memory dependence in MemProof. By testing each question under blind, recent-only, and evidence-revealed conditions, MemProof measures memory demand directly rather than inferring it from question wording, enabling a more reliable evaluation of long-term video memory (s1Jm, guab).
  • Improving memory without updating model parameters. EvoHarness improves memory organization, retrieval, and presentation through modular and interpretable program-level changes, while keeping all neural components frozen (s1Jm, guab, r36N).

We have responded to every weakness and question in the individual replies below, with new experiments where needed. If anything remains unclear, or if any part of our responses raises further questions, please let us know; we would be glad to discuss it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.