- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a training-free framework for streaming video understanding called TRACE. It retains potentially useful visual information for the future through a limited KV Cache budget. The paper designed a scoring mechanism to rate the visual Tokens, retaining the important visual content and reorganizing their position encoding, achieving effective Token compression. It has achieved excellent performance on benchmarks such as StreamingBench and OVO-Bench.
Critical strengths
- The research motivation of this paper is meaningful. It aims to solve the problem in the streaming video understanding scenarios: when future queries are unknown, how to retain effective historical information with a fixed budget, and the method should be without the need for training and can be used directly.
Critical weaknesses
- The method proposed in the paper retains the recent frames. As mentioned in study [1], StreamingBench and OVO-Bench can achieve high scores simply by relying on the most recent few frames. Therefore, the improvement reported in the paper on the benchmark test for streaming video understanding is likely due to the retention of recent frames. However, this method has shown a notable decline in performance when applied to long video understanding tasks. Thus, the validity of this method needs to be further verified.
[1] A Simple Baseline for Streaming Video Understanding
Questions for authors
- In the ablation experiment, using only short-term memory was able to achieve performance close to that of the complete method (OVO-Bench). Can this further prove the effectiveness of memory?
- Could the author provide a further explanation of the fundamental principles behind the STAR scoring mechanism?
- Does the Top-K selection process lead to the dominance of high-score historical markers in memory, thereby preventing new information from being retained? It would be helpful if the author could provide an explanation.
6 comments
Q2: Principle of STAR Scoring
Q2: Could the author provide a further explanation of the fundamental principles behind the STAR scoring mechanism?
Response: The principle behind STAR is query-agnostic evidence retention under unknown future queries. Since future query relevance cannot be estimated when retention decisions are made, STAR instead favors visual states that contribute distinctive information to the observed stream.
(1) Temporal and spatial novelty.
Temporal novelty measures how much a visual state changes relative to the same spatial location in adjacent observations, capturing newly introduced temporal evidence. Spatial distinctiveness measures how different a token is from its nearest available neighbors within the same frame, capturing localized information that differs from its spatial context.
These signals are complementary: a temporally stable token can still be retained if it contains spatially distinctive content, while a spatially subtle token can remain important if it captures a strong temporal change.
(2) Joint retention rather than sequential pruning.
STAR independently rank-normalizes the temporal and spatial scores and combines them before any token is removed:
The joint decision prevents either dimension from independently discarding evidence that is informative along the other. Top-K selection then realizes this principle under the fixed historical-memory budget.
STAR therefore does not attempt to predict future query relevance. Instead, it provides a lightweight query-agnostic criterion for retaining informative visual evidence before its future utility is known. We have clarified this principle in the revised manuscript (Sec. 3.3-3.4).
Q3: Adaptivity of Top-K Memory Updates
Q3: Does the Top-K selection process lead to the dominance of high-score historical markers in memory, thereby preventing new information from being retained?
Response: No historical token is permanently locked into memory. Top-K selection is recomputed over both previously retained and newly arriving candidates at every historical-memory update.
(1) New observations first enter short-term memory.
Every incoming frame is initially preserved at full granularity in the short-term observation window, independent of historical scores. Thus, previously retained tokens cannot prevent new observations from being accessed or evaluated.
(2) New and historical tokens compete jointly.
When frame leaves the short-term window, its tokens are combined with the currently retained historical tokens to form the candidate set:
STAR then evaluates the entire candidate set and selects the current Top-K tokens. Spatial distinctiveness is recomputed under the updated memory state, and both temporal and spatial scores are re-ranked before selection. Therefore, previous retention does not guarantee future retention: newly arriving tokens can replace older ones whenever they obtain higher joint scores.
High-scoring historical tokens may persist only if they remain competitive relative to newly arriving evidence. This persistence is intentional, as such tokens continue to represent distinctive historical information. We have clarified this adaptive update mechanism in the revised manuscript (Sec. 3.2, Appendix B.1).
W1 & Q1: Effectiveness of Historical Memory
Response: We agree that recent observations are strong for real-time perception, consistent with the observation in A Simple Baseline for Streaming Video Understanding. However, this does not imply that historical memory is unnecessary. Real-time perception and historical retrieval rely on the two memory components differently.
(1) Matched memory-component ablation.
In the original ablation, all variants use the same total budget C=5120, so the short-term-only variant can allocate the entire budget to recent frames and therefore covers a larger recent window than full TRACE. To isolate the contribution of each component, we additionally conducted a matched-component ablation: the short-term-only variant keeps the same N-frame allocation as full TRACE, while the historical-only variant keeps the same historical-memory budget, without reallocating the removed component's capacity.
Table I: Matched Ablation of Short-Term and Historical Memory.
The results reveal a clear division of roles. Short-term memory is particularly useful for real-time perception, whereas historical memory is more important for backward tracing: removing historical memory reduces OVO-BT from 50.7 to 46.8, while historical-only reaches 50.4, close to the full model. The full model performs best across all tasks, showing that recent full-granularity observations and sparse historical evidence are complementary rather than redundant.
(2) Effectiveness of novelty-guided historical retention.
We further isolate the contribution of STAR by keeping the TRACE memory architecture unchanged and replacing only STAR-based historical-token selection with uniform or random selection. All other settings remain identical.
Table II: Ablation of Historical-Token Selection within TRACE.
STAR improves over uniform selection by 6.0 points on OVO-RT and 6.2 points on OVO-BT, and over random selection by 6.3 and 9.6 points, respectively. This shows that the gains do not come merely from retaining recent frames; how historical evidence is selected also matters.
(3) Long-video performance.
We agree that TRACE incurs an accuracy drop relative to full-context backbone inference on long-video benchmarks. This reflects the intended memory–accuracy trade-off: TRACE retains only 8.2% of the original visual tokens while preserving 91.1% of the corresponding backbone performance on MLVU and VideoMME. Our goal is not to outperform full-context offline inference, but to preserve useful historical evidence under a bounded memory budget. This trade-off is distinct from the question of whether historical memory contributes under the same bounded-memory setting, which is directly addressed by the matched ablations above.
We have revised the manuscript to clarify the distinct roles of recent observations and historical evidence retention (Sec. 4.3, Appendix C.6).