- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies long-term memory in streaming video from two perspectives. It introduces MemProof to explicitly verify whether questions truly depend on historical visual evidence, and EvoHarness to iteratively improve external memory organization, retrieval, and presentation while keeping neural models frozen. The experiments show that strong performance on conventional streaming benchmarks does not necessarily imply effective memory use, while EvoHarness achieves clear gains on MemProof.
Critical strengths
-
The motivation is compelling and timely. The paper clearly distinguishes between questions that merely refer to the past and those that genuinely require historical evidence, exposing an important evaluation gap in current streaming-video benchmarks.
-
The design of EvoHarness is also pretty interesting. Instead of updating model parameters, it improves the external memory pipeline through failure diagnosis and validated program-level changes, making the optimization process relatively interpretable.
Critical weaknesses
- The practical efficiency of EvoHarness in a true streaming setting needs to be evaluated. EvoHarness involves multiple stages, while the paper reports the number of retrieved frames, it does not provide latency, throughput, memory usage, or a runtime breakdown, making its practical suitability for real-time streaming difficult to assess.
Questions for authors
-
After all benchmark construction, filtering, and data splitting steps, was the finalized MemProof benchmark independently re-run through the full blind/recent/evidence and multi-event verification pipeline?
-
Could you report the practical inference efficiency of EvoHarness, including latency, throughput, GPU memory, and ideally a per-module runtime breakdown?
-
What is the main deployment-time computational bottleneck in EvoHarness? Have you considered incorporating latency or computational cost into the harness-evolution objective so that the system jointly optimizes accuracy and efficiency?
4 comments
Author response to (Q1) Re-certification of the finalized benchmark.
(Q1) Re-certification of the finalized benchmark. We thank the reviewer for the suggestion. In short, the final benchmark had not been re-certified as a whole before submission, so we have now re-run the full verification on the final test questions.
Setup. Recall we verify each question truly needs the past in three stages: blind, recent, and evidence (main Sec. 2.2). A question "passes" only if the verifiers fail in the first two stages but succeed in the third. Multi-event questions must also pass the multi-event tests (main Sec. 2.3). These checks, however, were run on candidate questions, before the final benchmark was assembled. During assembly, some questions were reworded and some had their options reshuffled, so a question that passed as a candidate is not guaranteed to pass in its final form. As the reviewer points out, re-certifying the final benchmark is therefore the right way to confirm that its questions still need the past. From 4,396 test questions, we excluded the 750 implicit-verification questions, (as they were not built with the three-stage verification) and performed re-certification on the remaining 3,646 questions. We took the final test questions exactly as released and passed each of them through the same checks again, as if it were a new candidate: the same two verifiers, the same prompts, and the same pass rules.
Results. Of the 3,646 re-certified test questions, 3,296 (90.4%) passed and 350 failed. (The 750 implicit-verification questions were not re-certified, as explained above, and are not counted in either number.) The failures have three main causes: (i) Some questions were reworded after their original check, and the new wording can change how the verifiers respond. (ii) The verifiers do not give exactly the same answers on every run, so questions close to a pass threshold can change outcome. (iii) Some multi-event questions are labeled as needing multiple past events (Wide), but in the re-check they could be answered with fewer of these events. They still cannot be answered without the past, so they remain memory questions; they only fall to a lower level (Narrow instead of Wide), which our rule counts as a failure. We will include this analysis in our final manuscript.
Author response to (Q3) Deployment bottleneck and cost-aware evolution.
(Q3) Deployment bottleneck and cost-aware evolution. The main bottleneck is the LLM calls, not the memory. Of the 1.57 s of an question (Tab. R4), the answer call (answer.py) takes 0.69 s, the planner call (planner.py) 0.41 s, and frame preparation on the CPU (packer.py) 0.41 s, while memory access takes less than 0.06 s. The planner is the only extra cost of the memory system, matching the 0.4 s gap to Uniform 28+4. Cost can thus be cut by lighter LLM use (e.g., a smaller planner), caching the prepared frames, or more GPUs.
Our current evolution does not optimize latency or computational cost; it focuses only on improving accuracy, since the acceptance rule keeps a proposal based on its accuracy gain alone (main Sec. 3.2). The only cost constraint is the fixed 32-frame budget (supp Sec. 3.1). Under this accuracy-only objective, frames per question happened to decrease (), but LLM calls per question increased (; supp Tab. 8). Incorporating cost is straightforward, since each receipt already records frames, calls, and tokens; for example, the acceptance rule could reject proposals that exceed a limit on LLM calls or latency, in the same way as the frame budget. Our current manuscript leaves out these experiments but will include it as future work.
Thanks for the rebuttal with detailed answers. Most of my concerns are addressed, and I have no further questions.
Author response to (W1,Q2) Practical inference efficiency in the streaming setting.
(W1,Q2) Practical inference efficiency in the streaming setting. We thank the reviewer for this suggestion. In short, H6 answers a question in about 1.5 s on a single GPU, about 0.5 s slower than uniform frame sampling. Accessing the memory itself takes less than 0.1 s; most of the time goes to the two LLM calls of the harness, the query planner (
planner.py) and the final answer (asnwer.py), and to preparing the selected frames for the model (packer.py: CPU job).Setup. We ran the pipeline of Fig. 2 of the main paper on one H100 GPU, with the same frozen models: Qwen3.5-9B as the planner and the agent f(⋅), and Qwen3-Embedding-4B as the text embedder. As timing on the full 4,396-question test split would take too long, we used 216 test questions sampled across the six dimensions. We compare H0, H3, and H6 with memory-free baseline: Uniform 28+4 (Tab. 5 of the supplement).
Latency and throughput. We report two measures in Tab. R3. Latency is the time to answer one question when it is the only request; we give the median (p50) and the 95th percentile (p95). Throughput is the number of questions answered per second when 8 or 16 requests arrive at the same time. H6 answers in 1.53 s at the median and 2.48 s at the 95th percentile, compared with 1.06 s and 1.99 s for Uniform 28+4. With 16 simultaneous requests, H6 answers 2.5 questions per second, against 3.0 for Uniform 28+4.
Tab. R3. Latency and throughput on one H100 GPU (216 test questions, Qwen3.5-9B). Latency: time to answer one question when it is the only request. Throughput: questions answered per second with 8 or 16 simultaneous requests.
Runtime per module (Tab. R4). We timed each program of Fig. 2 of the main paper. For H6, the LLM call of
answer.pytakes 0.69 s, the LLM call ofplanner.py0.41 s, and preparing the selected frames for the model 0.41 s. Frame preparation inpacker.pyruns on the CPU, not the GPU: each selected frame is loaded from disk, resized, and encoded into the request sent to the model. Its cost therefore grows with the number of frames delivered to the model.retriever.py, which embeds the queries and matches them against the MacroEvents, takes 0.05 s, and reading the memory built bybuilder.pytakes 2 ms.Tab. R4. Runtime per module (mean seconds per question, one request at a time). Programs follow Fig. 2 of the main paper. Frame preparation runs on a CPU that was shared with other jobs during the measurement. Uniform 28+4 does not use these programs, so only its total time is shown.
builder.py: reading the memoryplanner.py: LLM callretriever.py: query embedding and matchingpacker.py: preparing the selected framesanswer.py: LLM call(s)GPU memory. The engine serving the planner and f(⋅) holds 17.7 GiB of weights; with the memory it reserves for serving requests, its peak use is 65.7 GiB of the 95 GiB card. The embedder adds 7.6 GiB of weights. The memory itself is a small text structure stored on disk, 47 KB per video on average.
Correction of the supplement. Sec. 4.7 of the supplement reports 5.6 s per question for H3 on OVO-Bench. These are average times per question in 'large batch' runs, in which about 32 requests shared 8 engines and waited for each other; they are not the latency of one question. Measured alone, H3 takes 1.54 s. We will revise it accordingly.