acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Novelty-Aware Spatiotemporal Evidence Retention for Streaming Video Understanding

10 views

Abstract

Streaming video understanding requires vision-language models to process evolving streams causally and preserve visual evidence for queries arriving at arbitrary times. However, models must make irreversible retention decisions under a bounded memory budget, without access to future observations or queries. Existing methods often treat temporal and spatial redundancy separately or rely on attention-based importance scores, potentially discarding useful evidence before complementary cues are jointly considered. To address this challenge, we propose TRACE, a training-free framework for novelty-aware spatiotemporal evidence retention. Following an observe–score–retain procedure, TRACE maintains a causal hierarchical visual memory that preserves recent observations at full granularity while characterizing visual tokens through two complementary query-agnostic signals: temporal novelty and spatial distinctiveness. Their joint evaluation enables adaptive evidence retention without relying on attention weights, while temporal gap-aware positional re-encoding supports long-horizon streaming inference. Extensive experiments across multiple VLM backbones demonstrate that TRACE achieves state-of-the-art performance among the compared open-source methods on streaming video understanding benchmarks, while preserving 91.1% of the corresponding backbone performance with only 8.2% of the visual tokens on long-video benchmarks. TRACE further maintains bounded visual KV memory and stable query latency over hour-scale streams.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Vision-Language Models, Visual Memory, KV Cache Management, Spatiotemporal Novelty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ptGKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training-free framework for streaming video understanding called TRACE. It retains potentially useful visual information for the future through a limited KV Cache budget. The paper designed a scoring mechanism to rate the visual Tokens, retaining the important visual content and reorganizing their position encoding, achieving effective Token compression. It has achieved excellent performance on benchmarks such as StreamingBench and OVO-Bench.

Critical strengths
  1. The research motivation of this paper is meaningful. It aims to solve the problem in the streaming video understanding scenarios: when future queries are unknown, how to retain effective historical information with a fixed budget, and the method should be without the need for training and can be used directly.
Critical weaknesses
  1. The method proposed in the paper retains the recent frames. As mentioned in study [1], StreamingBench and OVO-Bench can achieve high scores simply by relying on the most recent few frames. Therefore, the improvement reported in the paper on the benchmark test for streaming video understanding is likely due to the retention of recent frames. However, this method has shown a notable decline in performance when applied to long video understanding tasks. Thus, the validity of this method needs to be further verified.

[1] A Simple Baseline for Streaming Video Understanding

Questions for authors
  1. In the ablation experiment, using only short-term memory was able to achieve performance close to that of the complete method (OVO-Bench). Can this further prove the effectiveness of memory?
  2. Could the author provide a further explanation of the fundamental principles behind the STAR scoring mechanism?
  3. Does the Top-K selection process lead to the dominance of high-score historical markers in memory, thereby preventing new information from being retained? It would be helpful if the author could provide an explanation.
6 comments
AuthorsOpenReview ↗

W1 & Q1: Effectiveness of Historical Memory

W1: The improvement on streaming benchmarks may mainly come from retaining recent frames, as recent-frame baselines can already perform strongly on StreamingBench and OVO-Bench. The method also shows a performance drop on long-video benchmarks.

Q1: In the ablation experiment, using only short-term memory achieves performance close to the complete method on OVO-Bench. Can this further prove the effectiveness of memory?

Response: We agree that recent observations are strong for real-time perception, consistent with the observation in A Simple Baseline for Streaming Video Understanding. However, this does not imply that historical memory is unnecessary. Real-time perception and historical retrieval rely on the two memory components differently.

(1) Matched memory-component ablation.
In the original ablation, all variants use the same total budget C=5120C=5120, so the short-term-only variant can allocate the entire budget to recent frames and therefore covers a larger recent window than full TRACE. To isolate the contribution of each component, we additionally conducted a matched-component ablation: the short-term-only variant keeps the same NN-frame allocation as full TRACE, while the historical-only variant keeps the same historical-memory budget, without reallocating the removed component's capacity.

Table I: Matched Ablation of Short-Term and Historical Memory.

MsM_sMhM_hOVO-RTOVO-BTMVBench (Val)VideoMME
✓×69.746.860.444.1
×✓67.950.463.256.9
✓✓71.550.764.958.1

The results reveal a clear division of roles. Short-term memory is particularly useful for real-time perception, whereas historical memory is more important for backward tracing: removing historical memory reduces OVO-BT from 50.7 to 46.8, while historical-only reaches 50.4, close to the full model. The full model performs best across all tasks, showing that recent full-granularity observations and sparse historical evidence are complementary rather than redundant.

(2) Effectiveness of novelty-guided historical retention.
We further isolate the contribution of STAR by keeping the TRACE memory architecture unchanged and replacing only STAR-based historical-token selection with uniform or random selection. All other settings remain identical.

Table II: Ablation of Historical-Token Selection within TRACE.

MethodOVO-RTOVO-BTMVBench (Val)VideoMME
+ Uniform65.544.562.152.1
+ Random65.241.161.952.2
+ STAR (Ours)71.550.764.958.1

STAR improves over uniform selection by 6.0 points on OVO-RT and 6.2 points on OVO-BT, and over random selection by 6.3 and 9.6 points, respectively. This shows that the gains do not come merely from retaining recent frames; how historical evidence is selected also matters.

(3) Long-video performance.
We agree that TRACE incurs an accuracy drop relative to full-context backbone inference on long-video benchmarks. This reflects the intended memory–accuracy trade-off: TRACE retains only 8.2% of the original visual tokens while preserving 91.1% of the corresponding backbone performance on MLVU and VideoMME. Our goal is not to outperform full-context offline inference, but to preserve useful historical evidence under a bounded memory budget. This trade-off is distinct from the question of whether historical memory contributes under the same bounded-memory setting, which is directly addressed by the matched ablations above.

We have revised the manuscript to clarify the distinct roles of recent observations and historical evidence retention (Sec. 4.3, Appendix C.6).

AuthorsOpenReview ↗

Q2: Principle of STAR Scoring

Q2: Could the author provide a further explanation of the fundamental principles behind the STAR scoring mechanism?

Response: The principle behind STAR is query-agnostic evidence retention under unknown future queries. Since future query relevance cannot be estimated when retention decisions are made, STAR instead favors visual states that contribute distinctive information to the observed stream.

(1) Temporal and spatial novelty.
Temporal novelty measures how much a visual state changes relative to the same spatial location in adjacent observations, capturing newly introduced temporal evidence. Spatial distinctiveness measures how different a token is from its nearest available neighbors within the same frame, capturing localized information that differs from its spatial context.

These signals are complementary: a temporally stable token can still be retained if it contains spatially distinctive content, while a spatially subtle token can remain important if it captures a strong temporal change.

(2) Joint retention rather than sequential pruning.
STAR independently rank-normalizes the temporal and spatial scores and combines them before any token is removed:

SSTARt(i)=αS~Tt(i)+(1−α)S~St(i)S_{\mathrm{STAR}}^t(i)=\alpha \widetilde{S}_T^t(i)+(1-\alpha)\widetilde{S}_S^t(i)

The joint decision prevents either dimension from independently discarding evidence that is informative along the other. Top-K selection then realizes this principle under the fixed historical-memory budget.

STAR therefore does not attempt to predict future query relevance. Instead, it provides a lightweight query-agnostic criterion for retaining informative visual evidence before its future utility is known. We have clarified this principle in the revised manuscript (Sec. 3.3-3.4).

AuthorsOpenReview ↗

Q3: Adaptivity of Top-K Memory Updates

Q3: Does the Top-K selection process lead to the dominance of high-score historical markers in memory, thereby preventing new information from being retained?

Response: No historical token is permanently locked into memory. Top-K selection is recomputed over both previously retained and newly arriving candidates at every historical-memory update.

(1) New observations first enter short-term memory.
Every incoming frame is initially preserved at full granularity in the short-term observation window, independent of historical scores. Thus, previously retained tokens cannot prevent new observations from being accessed or evaluated.

(2) New and historical tokens compete jointly.
When frame fτf_\tau leaves the short-term window, its tokens OτO_\tau are combined with the currently retained historical tokens It−1I_{t-1} to form the candidate set:

Ct=It−1∪OτC_t = I_{t-1} \cup O_\tau

STAR then evaluates the entire candidate set and selects the current Top-K tokens. Spatial distinctiveness is recomputed under the updated memory state, and both temporal and spatial scores are re-ranked before selection. Therefore, previous retention does not guarantee future retention: newly arriving tokens can replace older ones whenever they obtain higher joint scores.

High-scoring historical tokens may persist only if they remain competitive relative to newly arriving evidence. This persistence is intentional, as such tokens continue to represent distinctive historical information. We have clarified this adaptive update mechanism in the revised manuscript (Sec. 3.2, Appendix B.1).

Reviewer_LvXQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work targets streaming video understanding. The challenge here is how to manage the visual memory to make sure a good efficiency-effectiveness balance. The method is training-free and driven by visual representation novelty in the temporal and spatial domains.

Critical strengths

The paper is trying to solve an urgent problem. VLMs are powerful, but due to their cost, they cannot be deployed in many real-world applications. The authors proposed TRACE to break this bottleneck.

Critical weaknesses
  1. The techniques to estimate temporal and spatial novelty are quite naive. In some work, such as SURGE from ICLR 2026, they specifically mentioned "global drift" when estimating visual feature novelty. So the question is: can a simple average-like scoring be solid enough to estimate feature-space novelty signals? The authors didn't prove the robustness of their novelty signal estimation in the paper.
  2. I think the comparison shouldn't be limited to streaming video understanding methods. The core of TRACE is using a novelty signal driven from the feature space to help manage visual memory. The existing work, including SURGE ICLR26, LongVU ICML25, STC CVPR26, etc., may perform better when doing the novelty computation. Like STC, it also uses feature-space difference to compute the novelty signal. Adding these methods into the comparison can help prove whether TRACE has a solid novelty estimation as well.
  3. It is not clear how the authors decide on those hyperparameters in their method. Please clarify this point.
  4. The baselines introduction is missing. Maybe it can be added to 4.1
  5. The experimental setting for video streaming is missing. Can the authors clarify it?
Questions for authors

All my questions are in the critical weaknesses section.

4 comments
AuthorsOpenReview ↗

W1: Robustness of Novelty Estimation

W1: The techniques to estimate temporal and spatial novelty are quite naive. SURGE specifically considers "global drift" when estimating visual feature novelty. Can a simple average-like scoring be sufficiently robust?

Response: We agree that coherent visual changes such as camera motion can inflate temporal differences. TRACE therefore does not rely on temporal novelty alone. STAR jointly evaluates temporal novelty and within-frame spatial distinctiveness, with the two signals independently rank-normalized before aggregation.

(1) Complementary spatiotemporal scoring.
Temporal novelty captures changes across neighboring observations, while spatial distinctiveness is computed within each frame. Thus, when temporal differences are affected by coherent motion, the spatial cue provides complementary evidence. Rank normalization further prevents one signal from dominating due to scale differences. We do not claim that this removes global drift; rather, it reduces reliance on temporal change alone.

(2) Robustness under camera motion and flicker.
To directly examine this issue, we additionally evaluated 100 medium-duration videos from VideoMME, comprising 300 QA pairs, with substantial natural camera motion, flicker, rapid editing, and other visual disturbances.

Table I: Robustness of Spatiotemporal Novelty Scoring.

MethodAccuracy (%)
Qwen2.5-VL-7B67.3
+ Temporal only62.3
+ Spatial only62.7
+ STAR (w/o normalization)64.3
+ STAR66.0

STAR improves over temporal-only scoring by 3.7 points, spatial-only scoring by 3.3 points, and joint scoring without normalization by 1.7 points. This supports the benefit of complementary spatiotemporal scoring and normalization when temporal differences are unreliable.

The lightweight design is intentional because retention scores must be repeatedly updated during streaming inference. We nevertheless acknowledge that severe coherent or adversarial visual changes can still affect novelty estimation, and we have clarified this limitation in the revised manuscript (Appendix E.1).

AuthorsOpenReview ↗

W2: Comparison with Feature-Space Compression Methods

W2: The comparison should not be limited to streaming video understanding methods. Existing work such as SURGE, LongVU, and STC may provide stronger feature-space novelty computation.

Response: We thank the reviewer for pointing out these closely related methods. We have expanded both the discussion and empirical comparison with LongVU, SURGE, and STC. These methods are related to TRACE in reducing visual redundancy, but differ in their target setting and retention mechanism.

  • LongVU is designed primarily for offline long-video understanding, where the video and current query are available during compression.
  • SURGE identifies informative feature changes for long-video compression and explicitly considers global feature drift; its setting is also primarily offline.
  • STC is the closest to TRACE in the streaming setting, as it performs temporal and spatial feature compression over continuously arriving visual inputs. TRACE differs by formulating retention as a query-agnostic evidence-selection problem and jointly evaluating temporal novelty and spatial distinctiveness before historical tokens are removed.

(1) Controlled comparison with STC.
Since STC provides the closest streaming setting, we compare STC with STAR within the TRACE framework under the same evaluation protocol across streaming and offline benchmarks.

MethodStream-RTOVO-RTOVO-BTMVBenchMLVUVideoMME
STC78.870.847.564.261.056.1
STAR79.771.550.764.962.158.1

TRACE consistently outperforms STC across all evaluated benchmarks. The improvement is particularly clear on OVO-Bench Backward Tracing (+3.2) and VideoMME (+2.0), while TRACE also improves StreamingBench (+0.9), OVO-RT (+0.7), MVBench (+0.7), and MLVU (+1.1). These results provide a controlled comparison showing that TRACE's joint spatiotemporal evidence retention is more effective than the temporal-spatial feature compression strategy used by STC under the evaluated setting.

(2) Reference comparison with LongVU and SURGE.
For LongVU and SURGE, we report their published results on overlapping long-video benchmarks. Because their backbones, compression protocols, and evaluation configurations are not fully identical to ours, we treat these results as a system-level reference comparison rather than a controlled comparison of compression criteria.

MethodMVBenchVideoMMEMLVU
LongVU66.960.665.4
SURGE--60.965.7
TRACE64.958.162.1

LongVU and SURGE obtain stronger absolute performance on some offline long-video benchmarks, which is expected given their offline compression setting and different system configurations. TRACE instead targets causal streaming retention when future queries are unknown, while maintaining bounded visual memory and supporting query-time access without future information. We therefore use these results to clarify the different operating regimes rather than to claim a controlled advantage over the offline methods.

We have added these related methods, the controlled STC comparison (Appendix E.4), and the distinction between offline compression and query-agnostic streaming retention to the revised manuscript (Sec. 2.1).

AuthorsOpenReview ↗

W3–W5: Hyperparameters, Baselines, and Streaming Protocol

W3: It is not clear how the authors decide on the hyperparameters in the method.

W4: The baseline introduction is missing.

W5: The experimental setting for video streaming is missing.

Response: We appreciate these comments. The relevant settings and several sensitivity studies are included in the original submission, but we agree that the selection rationale and overall streaming protocol should have been presented more explicitly.

(1) Hyperparameter selection.
TRACE uses one fixed configuration across datasets and backbones rather than tuning hyperparameters separately for individual benchmarks. Several key choices are already analyzed in the original submission:

  • Temporal/spatial weight α\alpha: Figure 4 studies the balance between temporal novelty and spatial distinctiveness. We use α=0.6\alpha=0.6 as the default and keep it fixed throughout all experiments.
  • Short-term window NN and total memory budget CC: Appendix Table 10 evaluates N∈{2,6,10}N\in\{2,6,10\} and C∈{2560,5120,10240}C\in\{2560,5120,10240\}. The default N=6N=6 and C=5120C=5120 achieves the highest average score (64.8) among the evaluated configurations.
  • Anchor layer: Appendix Table 8 compares early, intermediate, and final transformer layers. The final layer achieves the highest overall average score (64.8) and is therefore used throughout.
  • Spatial neighborhood: Appendix Table 9 compares adjacent and nearest-available spatial neighbors; the nearest-available strategy obtains the stronger overall result.
  • TGAP trigger: We set the trigger to 120K accumulated visual positions, slightly below Qwen2.5-VL-7B's configured maximum positional range of 128K. This is a model-capacity-driven choice that leaves a margin before reaching the positional limit, rather than a benchmark-specific tuned hyperparameter.

We have consolidated these choices and their corresponding analyses in Sec. 4.1 for clarity.

(2) Baseline descriptions.
We agree that Sec. 4.1 should explicitly introduce the evaluated baselines rather than relying mainly on the result tables. We have added a concise description grouping them into: (i) offline long-video VLMs, (ii) online/streaming VLMs, and (iii) within-backbone visual-memory/compression methods. Following W2, we also explicitly discuss LongVU, SURGE, and STC.

(3) Streaming evaluation protocol.
We have made the streaming protocol explicit. Frames are processed causally and sequentially at 1 fps. At each streaming step, TRACE updates its visual memory using only observations available up to that point; future frames and future queries are unavailable during retention. When a query is issued, the model answers using the currently retained visual memory without accessing future observations.

For implementation, we use up to 256 visual tokens per frame for the streaming benchmarks and 64 for the long-video benchmarks, a total visual KV capacity of 5,120 tokens, a six-frame short-term observation window, α=0.6\alpha=0.6, and the final transformer layer as the scoring anchor. TGAP is triggered at 120K accumulated visual positions. All experiments are conducted on NVIDIA RTX 4090 GPUs.

We have revised Sec. 4.1 to present the baseline descriptions, fixed hyperparameter configuration, and streaming evaluation protocol together so that the experimental setting is clear and reproducible.

Thank you for the further clarification. We respectfully disagree that the current evidence is insufficient to support our design. The original concern asked whether the lightweight temporal-spatial score is reliable under factors such as camera motion and global drift. In response, we added a controlled robustness study on 100 VideoMME videos with camera motion, flicker, rapid editing, and other visual changes. The full STAR design consistently outperforms temporal-only, spatial-only, and unnormalized variants. We also added a controlled comparison with STC under the same backbone and evaluation protocol, where TRACE improves performance across all evaluated benchmarks. These experiments provide direct evidence for the effectiveness of the proposed design.

We also want to clarify that the lightweight score is intentional. TRACE targets streaming video understanding, where memory must be updated whenever new visual inputs arrive. More complex global matching or repeated long-range comparison would add computation and memory access at every streaming step. This conflicts with the goal of fast processing and bounded memory. Our goal is therefore not to design the most complex novelty estimator, but to use lightweight cues that work well within an evolving historical memory.

We do not claim that the current score models every possible source of feature variation. This is a limitation, and we have made it clear in the revision. However, we believe that requiring separate validation for every possible feature-space disturbance goes beyond the original robustness concern. The added robustness study, controlled STC comparison, cross-backbone results, and efficiency analysis together provide sufficient empirical support for TRACE in its intended streaming setting.

We appreciate the reviewer’s feedback and the discussion throughout the rebuttal.

Reviewer_T59pOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the problem of video understanding in a streamed video setting, where vision-language models are required to answer queries about the current video stream so they have to retain information about the video. Because the video may be long and devices are often memory limited, models cannot hold the KV cache infinitely. The paper proposes keeping only the Top-K novel key value tokens. To this end, the paper proposes a score that prioritizes local differences in post-RoPE keys, combining temporal novelty (distance to the same spatial position in adjacent frames) and spatial distinctiveness (distance to neighboring tokens within the same frame). The evaluation is done on Qwen-VL backbones with OVO-Bench and StreamingBench. It shows that TRACE keeps GPU memory and latency bounded over hour-long streams.

Critical strengths

The design requires no training and does not rely on attention weights. It also reuses keys already in the KV cache, which keeps the additional overhead low.

The design uses constant amount of GPU memory regardless of the length of the video.

Critical weaknesses
  1. The novelty premise is not justified. The paper assumes that locally novel tokens should be retained, but it never shows that local novelty correlates with what future queries need. Figure 1 shows only that attention and novelty differ, not that novelty is the right criterion. Redundancy removal keeps one representative of repeated content, whereas novelty scoring can discard an entire static segment. It is not very robust. For example, white noise, flicker, or camera motion would inflate the scores.

  2. Redundancy removal and selecting locally novel tokens are different concepts.

  3. Key related work is missing. InfiniPot-V (NeurIPS 2025), a training-free, query-agnostic KV retention method under the same setting, is neither cited nor compared. ProtoKV (ICML 2026) groups similar KV tokens and stores centroids with residual statistics. This work explicitly argues that token retention is brittle under delayed queries. ReKV is cited but not compared. The paper does not explain why its approach should outperform these alternatives.

  4. Irreversible deletion risks losing history. Unlike ProtoKV, which absorbs evicted tokens into summaries, TRACE permanently deletes every token outside the global top-K. In a camera streaming case, an adversary can actually attack the system by repeatedly creating new signals (such as flickering) and in edited video case, some actually introduces highly dynamic "highlights" in the intro. The design in inherently brittle.

Questions for authors

How robust is the key-space novelty score to noise, flicker, and camera motion, and how does it compare with pixel-level or encoder-level novelty? Use youtube videos that are professionally edited.

The post-RoPE key's L2 distance is used to compute the score, which includes the position. The position itself doesn't have to do anything with the actual pixel value. So what's the motivation? What's the result if you use the original key before RoPE?

Is there direct evidence that novelty-selected tokens cover query-relevant evidence? For example, what is the evidence retention rate compared with random and uniform selection?

How does TRACE perform against random and uniform-subsampling baselines under the same budget?

Can the authors compare against InfiniPot-V, ProtoKV, and ReKV under matched settings? Can you also include a delayed-query evaluation?

8 comments
AuthorsOpenReview ↗

W1 & Q1/Q3/Q4: Validity and Robustness of Novelty-Based Retention

W1: The novelty-based premise may not always hold. Static but important content could be discarded, while noise, flicker, or camera motion may artificially increase novelty.

Q1: How robust is TRACE to such visual disturbances? How does key-level novelty compare with pixel- or encoder-level novelty?

Q3: Can the authors provide further evidence that TRACE preserves useful historical evidence?

Q4: How does TRACE compare with random or uniform retention under the same memory budget?

Response: We agree that visual novelty is not an oracle for future query relevance. TRACE instead uses novelty as a query-agnostic proxy for evidence diversity when future queries are unknown. A visual state is favored when it contributes temporal or spatial information that is less redundant with the observed stream. We therefore evaluate this premise from three complementary perspectives: robustness, budget-matched sampling baselines, and historical-evidence retention.

(1) Robustness to visual disturbances and scoring representation.
We evaluated 100 medium-duration videos from VideoMME, comprising 300 QA pairs with substantial natural camera motion, flicker, rapid editing, and other visual disturbances. We compare novelty computed from raw pixels, encoder features, and the key representations used by TRACE.

Table I: Robustness of Different Novelty Representations.

MethodAccuracy (%)
Qwen2.5-VL-7B67.3
+ Pixel-level novelty63.3
+ Encoder-level novelty62.7
+ Key-level TRACE66.0

Key-level TRACE remains closest to the uncompressed backbone and outperforms both pixel- and encoder-level novelty. This suggests that contextualized key representations provide a more reliable signal than low-level appearance changes under naturally occurring visual disturbances. We do not claim that TRACE is invariant to severe or adversarial disturbances. Persistent flicker or globally coherent changes can still affect novelty estimation. We have made this limitation explicit in the revised manuscript.

(2) Budget-matched uniform and random retention.
To separate novelty-guided retention from the benefit of simply keeping a fixed number of tokens, we apply direct uniform and random retention to the backbone visual tokens under the same total memory budget.

Table II: Budget-Matched Sampling Baselines.

MethodStream-RTOVO-RTOVO-BTMVBenchVideoMMEMLVU
Uniform70.855.240.362.552.455.9
Random66.949.239.062.853.855.8
TRACE79.771.550.764.958.162.1

TRACE consistently outperforms both sampling baselines, including +16.3 points over uniform retention on OVO-RT and +10.4 points on OVO-BT. Thus, the gains cannot be explained by the memory budget alone.

(3) Historical-evidence retention.
We further examine OVO-Bench subsets that specifically depend on retrieving earlier visual evidence. Under matched settings, TRACE substantially improves EPM and ASI compared with uniform and random retention.

Table III: Historical-Evidence Retention on OVO-Bench.

MethodASIEPMAvg.
Qwen2.5-VL-7B59.549.552.8
+ Uniform62.248.252.8
+ Random59.542.147.9
+ TRACE67.656.660.2

These results provide task-level evidence that TRACE retains historical information useful for later queries. They also clarify the intended role of novelty: TRACE does not predict future relevance directly, but preserves query-agnostic evidence diversity more effectively than simple sampling under the same bounded memory budget.

We have revised the manuscript (Appendix E.1-E.2, E.4) to clarify this interpretation and the robustness limitations of novelty-based retention.

AuthorsOpenReview ↗

W2 & W4: Novelty-Based Retention and Hard-Retention Trade-off

W2: Redundancy removal and selecting locally novel tokens are different concepts.

W4: Irreversible deletion risks losing history. Unlike ProtoKV, which absorbs evicted tokens into summaries, TRACE permanently deletes every token outside the global Top-K.

Response: We agree with both points. Novelty-based retention is not equivalent to redundancy removal, and hard retention represents a different trade-off from summary-state memory.

(1) Redundancy removal vs. novelty-based retention.
Redundancy removal asks which visual observations are repeated, whereas novelty-based retention asks which visual states contribute distinct evidence and should remain under a bounded memory budget. The two objectives overlap but are not equivalent. In particular, removing redundancy alone cannot keep memory bounded when distinct historical observations continue to accumulate.

TRACE therefore performs selective evidence retention, rather than pure redundancy removal. Temporal novelty and spatial distinctiveness are used to prioritize visual states that contribute distinct evidence; redundancy reduction is a consequence of this selection, not the objective itself. A temporally stable token can still remain competitive when it contains spatially distinctive information.

(2) Hard retention vs. summary-state memory.
We also agree that irreversible retention and summary-based memory make different design choices. ProtoKV compresses evicted history into summary states, whereas TRACE retains a bounded subset of the original visual KV states without constructing additional summary representations.

TRACE does not claim that hard deletion is universally preferable. Its goal is to improve which native visual states survive when memory must remain bounded, while keeping memory updates lightweight and preserving direct access to the retained KV representations.

We have revised the manuscript (Sec. 2.2) to clearly distinguish novelty-based retention from redundancy removal and to explicitly discuss the trade-off between hard retention and summary-state memory.

AuthorsOpenReview ↗

Q2: Pre-RoPE vs. Post-RoPE Key Scoring

Q2: The post-RoPE key's L2 distance includes position, which does not necessarily relate to the pixel value. What is the motivation, and what happens if the original key before RoPE is used?

Response: We agree that post-RoPE keys contain positional information. Importantly, the original submission already includes a controlled comparison between pre-RoPE and post-RoPE key scoring in Appendix C.1, Table 6.

Table IV: Pre-RoPE vs. Post-RoPE Key Scoring.

RepresentationOVO-BenchMVBenchVideoMMEAvg.
Pre-RoPE Key71.364.957.764.6
Post-RoPE Key71.564.958.164.8

The two representations perform almost identically (64.6 vs. 64.8 average). This shows that TRACE's effectiveness does not depend on the positional component introduced by RoPE.

We use post-RoPE keys by default because they achieve the highest overall score and can be directly reused from the cached key representations maintained during streaming inference. We have clarified this motivation in the revised manuscript (Appendix C.1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Key Clarifications and Additional Evidence

We sincerely thank the reviewers for their constructive feedback and recognition of TRACE's strengths, including its meaningful and urgent problem setting (LvXQ, ptGK), training-free and attention-weight-free retention design (T59p, ptGK), low additional overhead through reuse of cached keys (T59p), and bounded GPU memory independent of video length (T59p). The reviewers also recognized the importance of historical evidence retention under a fixed budget when future queries are unknown (ptGK). Below, we summarize the main clarifications and additional evidence. Detailed responses are provided in the reviewer-specific comments.

Q1: Validity and Robustness of Query-Agnostic Novelty Retention (T59p, LvXQ).

TRACE does not assume that visual novelty directly predicts future-query relevance. When future queries are unavailable at retention time, TRACE formulates streaming memory as query-agnostic evidence retention under a bounded budget, using temporal novelty and spatial distinctiveness as complementary signals. We added robustness experiments on 100 medium-duration VideoMME videos with 300 QA pairs containing natural camera motion, flicker, and rapid editing. Key-space novelty achieves 66.0%, compared with 63.3% for pixel-level and 62.7% for encoder-level novelty, remaining closest to the uncompressed backbone (67.3%). Joint STAR scoring (66.0%) also outperforms temporal-only (62.3%), spatial-only (62.7%), and joint scoring without normalization (64.3%). These results support the robustness of complementary spatiotemporal scoring under naturally occurring visual disturbances.

Q2: Effectiveness of Historical Memory Beyond Recent Frames (T59p, ptGK).

We added a matched memory-component ablation that fixes the allocation of short-term and historical memory. On OVO-Bench, short-term-only achieves 69.7/46.8 on Real-Time/Backward Tracing, historical-only achieves 67.9/50.4, while full TRACE reaches 71.5/50.7. This reveals a clear division of roles: recent observations are especially useful for real-time perception, whereas historical memory is more important for backward tracing. We further isolate STAR by keeping the TRACE memory architecture unchanged and replacing only historical-token selection. STAR reaches 71.5/50.7 on OVO-RT/BT, compared with 65.5/44.5 for uniform and 65.2/41.1 for random selection. Thus, the gains cannot be explained merely by retaining recent observations; how historical evidence is selected also matters. We also clarify the long-video results as a memory–accuracy trade-off: TRACE retains only 8.2% of the original visual tokens while preserving 91.1% of the corresponding backbone performance on MLVU and VideoMME.

Q3: Comparisons, Streaming Protocol, and Reproducibility (T59p, LvXQ).

We expanded the comparisons to include InfiniPot-V, ProtoKV, ReKV, LongVU, SURGE, and STC. For STC, the closest streaming feature-space compression baseline, we added a controlled comparison under the same backbone and evaluation protocol; TRACE consistently outperforms STC, including +3.2 on OVO-Bench Backward Tracing and +2.0 on VideoMME. We further clarify that the distinction from STC is not the use of temporal and spatial cues. STC uses historical context to guide compression of the current-frame tokens. In contrast, TRACE maintains an evolving bounded historical memory in which previously retained and outgoing visual states are jointly reconsidered under the same persistent capacity. We also clarified the streaming protocol and hyperparameter settings: TRACE processes frames causally without access to future frames or queries and uses fixed core memory hyperparameters across datasets and backbones.

We have revised the manuscript accordingly, with clearer descriptions of the streaming protocol and experimental settings (Sec. 4.1), and additional comparisons and analyses supporting the main claims (Appendix E.1--E.4). Overall, the revisions strengthen the central claim of TRACE: when future queries are unknown, maintaining an evolving historical memory in which old and new visual evidence continually compete under a bounded capacity provides an effective and lightweight strategy for persistent evidence retention in streaming video understanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.