- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to use memory effectively in streaming video understanding. Rather than introducing another complex memory architecture, the authors first examine three design choices: how to access past information, what evidence to present to the model, and when to retrieve additional history.
Based on these observations, they propose TRACE, a training-free framework that progressively expands the context from recent video frames to sparse short-term memory and, when needed, retrieved historical events. The decision to expand is based on the probability gap between the two most likely predicted tokens. Historical events are indexed using captions but presented to the model as visual frames with temporal information. TRACE also reuses KV caches across stages to avoid redundant computation.
On OVO-Bench, TRACE improves Qwen3.5-9B from 63.49% to 72.62% overall accuracy, while reducing per-query inference latency from 19.10 s to 1.86 s. The authors also report improvements on StreamingBench and with other VLM backbones.
Critical strengths
- The comparisons of memory access strategies, visual evidence, and temporal information help explain why adding more history can reduce accuracy.
- Without additional training, TRACE improves Qwen3.5-9B by 9.13 percentage points on OVO-Bench while substantially reducing query-time latency.
Critical weaknesses
-
The benefit of the intermediate SM stage needs clearer evidence. Most components of TRACE have already been explored in prior work, particularly OASIS. Table 8 compares access strategies within TRACE, but does not fully separate the benefit of three-stage access from differences between TRACE and earlier approaches.
-
TRACE gains 1.91 points over SimpleStream at roughly three times the query latency and FLOPs. Table 8 also shows gains of 0.62 and 1.16 points from successive memory expansions. Results grouped by historical evidence distance, together with statistical tests, would help assess whether these gains justify the extra cost.
-
High-confidence errors may prevent retrieval. TRACE uses the top-two token probability margin to decide whether to stop, but the paper does not report how often it stops with an incorrect answer when relevant historical evidence is missing from the current context.
-
The reported latency differs between Section 4.2 and Table 5: 2.31 s versus 1.86 s. Separately, some analyses use different context configurations and evaluation protocols, making direct comparisons difficult.
Questions for authors
-
OASIS already uses uncertainty-triggered historical retrieval. Could the authors compare it with TRACE's three-stage strategy under the same memory representation, retriever, and computational budget? This would help clarify what the additional SM stage contributes.
-
How often does TRACE stop at RC or SM with a high-confidence incorrect answer, even when the correct evidence is available in long-term memory? How much do these cases affect overall accuracy?
-
The reported latency excludes memory construction and updates. How does TRACE compare with SimpleStream and OASIS when these costs are included, especially at different query frequencies?
Response To Reviewer PAbZ
Thank the reviewer for the constructive feedback.
1. (W1, Q1) Contribution of the Intermediate SM Stage
We agree that uncertainty-triggered retrieval has been explored in prior work, including OASIS. TRACE instead analyzes complementary historical evidence and integrates it through progressive context expansion.
Table 8 compares access strategies with identical backbone, memory, and retrieval:
Direct RC → RM reduces Overall by 0.54 points versus RC only. Adding SM instead improves Overall by 0.62 points and Backward by 2.06 points, allowing some queries to be answered using sparse visual context and summaries. The full pipeline improves Overall by another 1.16 points over RC → SM, and by 2.32 points over RC → RM (Backward: +4.75 points). SM thus provides a complementary evidence layer, not merely extra computation.
2. (W2) Accuracy–Efficiency Trade-off and Evidence Distance
We acknowledge that TRACE requires more query-time computation than SimpleStream-4. TRACE prioritizes historical reasoning while preserving real-time perception: compared with SimpleStream-4, it trails Real-Time by 0.35 pp (82.92 vs. 83.27), but gains 4.91 pp on Backward (58.95 vs. 54.04) and 1.91 pp Overall (72.62 vs. 70.71).
Since OVO-Bench lacks complete evidence timestamps, we manually annotated 100 questions and grouped them by query-to-evidence distance d:
SimpleStream performs best for near-term evidence (≤10s), where retrieval may be unnecessary. For more distant evidence, TRACE exceeds SimpleStream by 12.90 points (10–30s) and 9.50 points (>30s), and OASIS by 3.23 points and 2.36 points, respectively. This exploratory result supports TRACE's advantage on distant evidence.
3. (W3, Q2) Overconfidence and Premature Stopping
We agree that the top-1/top-2 probability margin cannot reliably detect every case of insufficient evidence. Overconfident errors can prematurely stop routing at RC or SM and miss relevant evidence.
In an additional question-level comparison across 1,468 questions, 191 answers changed from incorrect to correct, while 130 changed from correct to incorrect. Historical access can both correct errors and introduce distracting evidence. TRACE therefore balances retrieval benefits against distraction, rather than maximizing invocation.
4. (W4) Latency and Evaluation Protocol
We apologize for the inconsistency. The correct TRACE (Qwen3.5-9B) query latency is 1.86s, as reported in the Abstract and Table 5; the 2.31s in Section 4.2 is outdated and will be corrected.
Section 2 conducts controlled analyses of Access, Information, and Invocation, sometimes using fixed contexts, golden evidence, or different evidence-presentation protocols to isolate individual factors. Section 4 instead evaluates the full progressive-routing pipeline under a strict online streaming protocol.
5. (Q3) Total Cost Across Query Frequencies
We agree that query latency alone does not capture total streaming-system cost. Appendix B.2 (Table 10) reports online memory construction: 27.8s/min of video for TRACE, versus 116.8s/min for OASIS.
Let q denote queries per minute of video. Combining average memory-maintenance and QA costs gives the following estimated cumulative compute time (seconds per minute of video):TRACE: 27.8+1.86q, OASIS: 116.8+13.84q, Vanilla: 19.10q, SimpleStream-4 / -16: 0.59q / 1.79q
TRACE is cheaper than OASIS across query frequencies. SimpleStream is cheaper still without persistent memory, but weaker on historical reasoning. Compared with Vanilla, TRACE reaches compute break-even at approximately 1.61 queries/min.
Memory construction is asynchronous and query-independent, running as video arrives. Hence the measured query-to-answer latency is 1.86s per query. Our efficiency claim concerns online responsiveness and accuracy–cost trade-offs.