- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies query-agnostic memory construction for streaming video understanding with frozen vision-language models. It represents a memory mechanism as a program with recent, episodic, and gist memory, and uses an LLM to propose and implement changes within a constrained DSL. Candidates are evaluated on NExT-GQA, and the selected program is subsequently tested on StreamingBench and OVO-Bench. The resulting system improves accuracy over the reported frozen-backbone baselines while substantially reducing reader context. The main question for this review is whether the evidence establishes the value of the proposed discovery procedure, beyond the effectiveness of the resulting memory configuration.
Critical strengths
The formulation makes a useful distinction between searching for a reusable memory-update program and optimizing memory contents for individual videos. The separation between query-independent updates and query-conditioned retrieval is appropriate for the task. Explicitly including bounded working state in Section 3.1 is also a sound design choice: bounding only the reader-facing memory would not suffice for a streaming system.
Searching on NExT-GQA and freezing the program before external benchmark evaluation is a meaningful strength. The improvements across two benchmarks and two Qwen backbones suggest that the selected configuration has practical value beyond its discovery dataset. Appendix A provides a helpful account of unsuccessful alternatives as well as successful changes, and Appendix D candidly illustrates how historical evidence can mislead current-scene understanding. These analyses are more informative than reporting only the final aggregate scores.
Critical weaknesses
The central contribution of LLM-guided discovery is not isolated experimentally. The three memory roles, packing order, and staged search schedule are prescribed, while many reported changes concern capacities, retrieval depths, thresholds, and segment lengths. The extensible operator interface could make this more than configuration tuning, but the paper does not establish how much that capability contributes. An improving trajectory over 89 candidates shows that search found a better configuration; it does not establish an advantage from semantic proposals, feedback-conditioned refinement, or operator invention. There is no budget-matched random or conventional configuration-search baseline, and no comparison with LLM proposals that do not receive experimental history. One informative control under the same evaluation budget, together with a clear account of which useful operators were newly implemented, would substantially strengthen the main claim.
The external comparisons do not sufficiently distinguish the benefit of the discovered mechanism from simpler context selection. In Figure 4a, recent memory alone scores 79.80, versus 81.40 for the complete system; this is a much smaller increment than the improvement from the 75.7 backbone result in Table 1. Moreover, R+G reaches 80.88, while R+E slightly underperforms R alone. These results support some value from historical memory, but provide limited aggregate support for the necessity of the selected episodic branch. The initial reference program is not evaluated on the external benchmarks, so the downstream benefit of discovery itself remains unclear. The same controls would be particularly useful on OVO-Bench BT, where historical retention is directly tested. Appendix B also states that baseline results are collected from prior papers, with different frame rates and incompletely aligned inference settings. Such comparisons provide context, but the small margins over the closest methods, such as 81.4 versus 81.3 on RTVU, should not be interpreted as established superiority without a matched protocol and an uncertainty estimate.
The efficiency evidence supports lower query-time reader cost more clearly than lower overall streaming cost. Section 4.3 explicitly profiles the reader, yet describes the latency reduction as end-to-end inference efficiency. Appendix C explains that the writer uses one foreground GPU for visual extraction and a second background GPU for summarization and consolidation. Its measured average processing time is already close to the one-second sampling interval. This is useful profiling, but it does not establish a resource-matched system-level advantage over the uniform-history baseline. In particular, Equation 2 permits flushing pending state at query arrival, and it is unclear whether the reported query latency includes this work and any unfinished writer jobs. A precise accounting of GPU allocation, writer compute, pending-state finalization, and query response time under an arriving stream would resolve the scope of the efficiency claim; faster reader inference alone remains a valid result.
Important details needed to reproduce the executable mechanism and its selection remain unspecified. For example, the paper names the selected coverage-change retention operator but does not define its scoring rule, and it calls the admission threshold a median without specifying how that median is estimated causally with bounded state. The numerical reader-context budget and the procedures for pooling and merging visual gist payloads are also unclear. These details determine the actual program behavior rather than merely implementation convenience. There are related ambiguities in the search record: Figure 2 and Appendix A.2 select episodic/gist retrieval depths of (8, 4), whereas Figure 3 displays (4, 8). Appendix A.1 reports identical scores for all nine coverage-retention variants but still chooses one specific variant, without an explicit tie-breaking rule or measurements explaining that choice. The paper should provide a consistent final configuration and enough operator and selection details to make the claimed discovery auditable.
Questions for authors
- Which operators were available in the initial library, and which were newly implemented by the agent? How much of the final improvement depends on the latter, compared with searching compositions and parameters of the initial library under the same budget?
- What sampling, frame cap, image resolution, prompting, and answer-scoring protocol produced the bare-backbone rows in Table 1? Under that same protocol, how do recent-only memory and the initial reference program perform on OVO-Bench BT?
- When a query arrives during an unfinished gist segment or a background consolidation job, what memory is made available to the reader, and is the time to finalize that snapshot included in the reported latency? How are reader and writer GPUs allocated in this measurement?
- Which retrieval depths were actually used for the external results? Why was the optical-flow retention variant selected among nine equal-scoring candidates, and does it change the stored or retrieved evidence on the discovery videos?