- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Sprout, an agentic framework that builds a temporal video-memory tree while answering questions. read_next provides coarse chronological coverage, write_chunk_memory stores textual memory nodes, and search_memory together with high-frame-rate revisits supplies targeted visual detail. The system removes video frames from the active conversation after recording textual observations, while retaining the memory tree and question–answer history for later questions. Across LVBench, Video-MME Long, Video-Holmes, and one EgoLifeQA recording, the reported results are competitive with or better than the listed memory baselines; the Gemini comparison also reports lower aggregate input/output tokens. The watch/remember/revisit abstraction is a useful systems contribution. The current evidence, however, does not yet determine how much of the accuracy gain comes from the proposed temporal memory mechanism rather than sequential question context, question order, amortization choices, or model/protocol differences.
Critical strengths
-
The paper presents a concrete interface for online memory construction: coarse chronological coverage, targeted high-rate revisits, and write-back into a shared temporal tree. This abstraction is well suited to repeated questions about long videos.
-
The evaluation spans three model families and several long-video settings, including LVBench, Video-MME Long, Video-Holmes, and EgoLifeQA. The frame-release, question-history, literal-versus-embedding retrieval, context-window, and token analyses make the system trade-offs more visible than a single end-task score would.
Critical weaknesses
-
The reported accuracy gains are not yet attributable to the proposed memory mechanism (pp. 7–9, Tables 1–4). Sprout answers questions sequentially and exposes earlier questions and answers to later turns (p. 7), but the manuscript does not state whether the reported order is the benchmark order or a randomized order. Direct answers each question independently, while Qwen-MM-Plugins uses a different memory construction and model stack. Table 4’s Causal variant also changes several factors together: the first question receives the whole video, and later questions receive earlier questions, reasoning, and answers as text; the variant changes Video-Holmes by +3.9 points. This design does not isolate the tree or revisit/write-back mechanism. Table 2 introduces a separate protocol mismatch for EgoLifeQA: the direct baseline uses Gemini 2.5 Pro, whereas Sprout uses Gemini 3.8 Flash, and the paper does not report a same-backbone direct control. Equal-history, same-backbone controls and question-order sensitivity are needed before the headline causal claim can be assessed.
-
The cost and reproducibility evidence does not yet support an independent comparison (p. 7 for implementation details; p. 9, Table 3, Appendix B). The reported offline construction cost is amortized over the benchmark question count, while Sprout’s cost depends on the number and order of prior questions. A per-video total and a break-even curve over the number of questions would make the comparison interpretable. Table 3 reports aggregate input/output token totals, but does not separate memory-writing and search calls, frame/image-token usage, wall-clock latency, model pricing, or monetary cost. The paper also reports single point estimates without uncertainty intervals and leaves important runtime details under-specified, including exact API model identifiers, decoding settings, retry and tool-failure handling, and the concrete search cap. Finally, Eq. (1) bounds the accumulated video-frame term; text memory, tool outputs, reasoning, and question history can still grow, so the corresponding context claim should be stated with that scope.
Questions for authors
-
Were Direct and Qwen-MM-Plugins evaluated with the same question order and the same prior question–answer history available to Sprout? Please provide equal-history results under several random question orders, or report how the gains change when history is removed.
-
For EgoLifeQA, can you report Direct and Qwen-MM-Plugins with the same Gemini 3.8 Flash and Qwen3.8-Max backbones, frame budgets, and prompts used by Sprout? Please state whether each external baseline in Table 1 was rerun under the same protocol or taken from its source.
-
How are the values in Table 3 computed? Please separate per-video construction and per-question answering costs, state whether tool schemas, summaries, retries, and embedding calls are included, and report the break-even cost as the number of questions per video varies.
-
What exact API model identifiers and access dates, decoding settings, retry and malformed-tool-call handling, search-count cap, round-cap behavior, and frame/token budgets were used for each model and baseline? These details are needed to interpret the single-run accuracy numbers.
-
Can you provide no-history, no-revisit, flat-memory, and fixed-budget controls, together with retrieval precision/recall or a failure analysis of missed transient details and incorrect write-backs? These controls would show whether the improvement comes from the proposed memory mechanism rather than from additional model interaction.
5 comments
Response to Reviewer 734v
-
Component contributions and qualitative analysis. (Weakness 1; Question 5)
Component contributions. We add Sprout's components step by step with Gemini 3.8 Flash (Table 5). The first setting only reads video chunks and writes memory, using
read_nextandwrite_chunk_memory. The full memory is included in context. The second addssearch_memoryand removes the full memory from context. The agent then accesses memory through search.Setting LVBench Acc (%) LVBench Tokens (K) Video-Holmes Acc (%) Video-Holmes Tokens (K) Read and write memory only 73.2 24.3 70.5 25.6 + Search memory on demand 73.8 35.6 71.7 31.4 + Node revisiting and write-back 87.5 86.2 72.6 88.6 + Time-interval revisiting and write-back 87.9 81.5 73.8 82.1 + Question history (full Sprout) 88.2 71.7 76.8 74.5 Node revisiting and write-back substantially improve LVBench accuracy. Adding time-interval revisiting and write-back improves both benchmarks while reducing token use. Question history brings further gains.
Qualitative analysis. Our case studies illustrate how memory search guides revisiting, and how revisiting recovers missing details or corrects errors in memory. In Appendix G (Fig. 10), Q4 retrieves the relevant event, but memory lacks the athlete's bib number. Revisiting recovers it. In Q15, Sprout revisits the result graphic and corrects a race time in memory from 10.08s to 10.09s before answering.
Appendix H (Fig. 11) shows how memory search helps locate the relevant interval. Qwen-MM-Plugins revisits 10:55–11:45 and incorrectly answers Usain Bolt. Sprout locates 08:40–09:10 and identifies Anna Rogowska. Together, these examples show how memory and revisiting work together to support accurate answers.
-
Scope of Eq. (1). (Weakness 2)
We have updated Eq. (1) in Section 3.3 () to state its scope explicitly: releasing frames bounds the visual term, while the textual context (prompt, question history, retrieved text, and per-round outputs) still grows across rounds and questions, with the memory tree itself kept outside the context.
We appreciate your time and thoughtful suggestions. Please let us know if any concerns remain; we would be happy to address them.
Friendly reminder from authors
Dear Reviewer 734v
Thank you again for your time and valuable feedback on our submission. As the discussion period is closing soon, we would like to gently check whether our rebuttal and the additional results have addressed your concerns. If any point remains unclear, we would be glad to respond promptly before the deadline.
Sincerely, Authors
Response to Reviewer 734v
Thank you for your careful review and for recognizing the value of online memory across questions. Your suggestions on fair comparisons and reproducibility helped us clarify the evaluation. Below, we present additional comparisons and ablations, along with details of our experimental setup.
Weakness / Question:
Performance without question history. (Weakness 1; Question 1)
To separate the effects of question history and memory reuse, we compare shared and reset trees, each with and without question history. Resetting clears memory before each question.
Sharing the tree reduces token consumption by reusing visual observations across questions, while question history provides additional context that can improve accuracy and further reduce token usage.
Same-backbone comparisons and baseline protocols. (Weakness 1; Question 2)
On EgoLifeQA, we ran Qwen-MM-Plugins with both Qwen3.8-Max and Gemini 3.8 Flash, using the same frame budgets as Sprout. We have also added same-backbone Direct results for both models (51.4% and 53.6%) to Table 2. Baseline results in Tables 1 and 2 are taken from the original papers, except for entries marked with
*, which come from our own reruns.Token accounting and memory construction cost. (Weakness 2; Question 3)
Token accounting. Table 3 sums input and output tokens over all rounds of each question's final retained run, including tool schemas, video frames, generated summaries, and retrieved text. Sprout's online memory writes and searches are included in this total, and its literal matching requires no embedding calls. For Qwen-MM-Plugins, offline node-embedding tokens are included in Construction, while query-time embedding calls are not counted in Answering. API retries outside the retained run are excluded.
Memory construction and break-even. For Qwen-MM-Plugins, offline construction averages 1,131K tokens per video on LVBench and 630K on Video-MME Long (75.2K and 210.0K per question in Table 3). On LVBench, Qwen-MM-Plugins' answering cost alone (95.3K) already exceeds Sprout's total cost (71.7K), so it never breaks even regardless of question count.
Model settings, tool limits, and error handling. (Weakness 2; Question 4)
Model settings. We summarize Sprout's API settings below. The dates refer to public releases or snapshots, not API access dates.
For Gemini 3.8 Flash, “—” indicates that the backend ignores temperature.
Tool limits and error handling. Experimental Setup lists the frame budgets and the 32-round limit. At this limit, the agent must answer using the available evidence, as described in Method. Each question allows at most eight memory searches. Invalid tool calls return an error message so the model can correct them in the next round. Failed API requests are retried at most five times.