acceptodds
Under review as a conference paper at ICLR 2027

Beyond Saliency: Token Memory via Marginal Contribution for Streaming Video Understanding

Abstract

Streaming video understanding requires processing an unbounded video stream within bounded memory while answering queries whose timing and content are unknown in advance. Because evicted tokens cannot be recovered, memory management should preserve information not already covered by the current memory rather than simply retain individually salient tokens. Existing saliency-based methods do not explicitly model such redundancy, while age-stratified methods constrain competition across temporal tiers. We propose SMART, a training-free, query-independent streaming token memory that retains tokens according to their marginal contribution beyond what the current memory already covers. SMART groups consecutive frames into content-based events to distinguish within-event redundancy from observations across events. Within each event, cross-frame nearest-match weights estimate how much a token is already covered by other retained frames, and modulate an intra-frame determinantal point process (DPP) that measures token complementarity. Because marginal contribution changes as memory evolves, SMART rescores surviving tokens whenever a new frame arrives. An age-augmented DPP kernel further combines contribution with recency for unprotected tokens, while preserving the most recent frame for real-time perception. With Qwen3-VL-8B, SMART achieves 68.6% OVO-Avg and 82.4% accuracy on StreamingBench, the highest aggregate results among the streaming methods evaluated.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
streaming video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gvnoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies fixed-capacity visual-token memory for query-independent streaming video question answering. Its training-free policy segments frames into causal content events, uses same-event nearest-neighbor residuals to modulate per-frame DPP greedy scores, and applies age decay while preserving recent frames. Experiments with Qwen2.5-VL-7B and Qwen3-VL-8B report the highest listed aggregate scores on OVO-Bench and StreamingBench, together with component-level ablations.

Critical strengths
  • Event-scoped cross-frame matching is a useful way to distinguish repetition within one continuous event from similar content observed at different times. Combining this with intra-frame DPP scores gives the policy a concrete mechanism for preserving both temporal novelty and spatial complementarity without training.

  • SMART has the highest listed OVO-Avg and StreamingBench accuracy with both tested backbones in Table 1. In the fixed-backbone results in Table 2, removing cross-frame deduplication lowers OVO-Avg by 1.3 points and StreamingBench by 1.1 points on both backbones. Global deduplication also lowers all four aggregate results, which supports restricting redundancy to the detected event.

Critical weaknesses
  • SMART can distinguish redundancy from a repeated occurrence only when the event labels are correct, but Equations (1)–(2) detect a new event using the median token similarity of the whole frame. A small but important action in an otherwise static scene may not change this median, so two separate occurrences can remain in the same event and be treated as duplicates. The global-deduplication ablation in Table 2 shows that event scoping helps, but Figures 6–8 mainly show scene or camera changes and do not test repeated actions, counting, or temporal order within the same scene.

  • Table 2 does not support the claim that marginal-contribution scoring is the largest source of improvement. Replacing sis_i with random values turns eviction into a noisy policy, so its 2.0–2.7 point drop cannot be compared directly with the cleaner component removals. A uniform-score age-only control and an si=wis_i=w_i control without the intra-frame DPP are needed to isolate what the proposed score adds.

  • Full-memory rescoring is presented as a central contribution, but its accuracy benefit is never isolated even though it dominates the cost. Table 4 reports 728 ms for greedy ordering and 739 ms for memory management, compared with 58 ms for visual encoding, while Table 2 has no frozen-score or current-event-only rescoring control. Without those controls or latency measurements for simpler retention baselines, the paper does not show that full rescoring provides a better accuracy–latency trade-off.

Questions for authors
  1. A few qualitative examples showing how Equations (1)–(2) segment repeated actions in the same scene would be helpful. Cases involving counting or temporal-order questions would clarify whether separate occurrences remain distinguishable in memory.

  2. Adding si=1s_i=1, si=wis_i=w_i, and frozen-at-insertion score controls to Table 2 would make the ablation more informative. Under the same age discount and protection settings, these rows would separate the effects of cross-frame novelty, intra-frame DPP, and dynamic rescoring.

  3. A matched latency comparison for SMART and the three controls above would also be useful. Reporting both per-frame update latency and end-to-end query latency would show whether full-memory rescoring is worth its 728 ms cost.

6 comments

Thank you for the constructive comments. We address each point below.

W1 & Q1: Separate occurrences within the same scene.

Two occurrences in the same event are not treated as duplicates merely because they share the event label, since SMART decides the scope of comparison and redundancy at two different levels. Event segmentation (Eqs. 1–2) operates on frames and only decides which retained frames serve as references, whereas redundancy is decided per token by its best match within that scope, wi=1−(max⁡j∈Cffi⊤fj)2w_i = 1-(\max_{j\in C_f} f_i^\top f_j)^2 (Eq. 5), not by the frame-level median.

A missed boundary therefore does not necessarily suppress a local change: tokens with low similarity to the retained references, such as those on a moving hand or a changed object, can still receive high novelty weights, while the unchanged background receives a low wiw_i, as intended. Global deduplication in Table 2 is the extreme case with no boundaries at all, and it lowers OVO-Avg/SBench by 0.7/1.3 (Qwen3-VL-8B) and 0.5/0.6 (Qwen2.5-VL-7B). Likewise, at θ=0.55\theta=0.55, where 90% of the windows contain a single event (Figure 4), OVO-Avg changes by about 0.2 points (Figure 3(b)). Overall accuracy therefore tolerates segmentation errors well.

For counting, which Q1 raises, the StreamingBench counting subtask (CT) covers both object and action counting. On both backbones, SMART has higher overall CT accuracy than both the age-only variant and the variant without cross-frame deduplication:

VariantQwen3-VL-8B CTQwen2.5-VL-7B CT
si≡1s_i\equiv1 (age only)40.338.3
w≡1w\equiv1 (no cross-frame deduplication)41.737.8
SMART43.140.9

The remaining case is a repetition that is nearly identical at the token level (same position, pose, and appearance) to a retained occurrence in the same event. The visual evidence itself is then repeated. We will add qualitative examples of repeated actions within the same scene, including counting and temporal-order cases, to the revised paper.

W2 & Q2: Variants of the scoring term.

The randomized variant keeps the age discount and the protection rule and replaces only the content score sis_i. To attribute the gain without relying on random scores, we add the deterministic si≡1s_i\equiv1 and si=wis_i=w_i variants requested in Q2, which also change only sis_i (the frozen-score variant is reported in the next part):

Table R1. Scoring variants (only sis_i differs).

sis_iInformation usedQwen3 OVO-AvgQwen3 SBenchQwen2.5 OVO-AvgQwen2.5 SBench
si≡1s_i\equiv1age only67.880.565.078.6
si=wis_i=w_icross-frame novelty only67.681.463.780.4
w≡1w\equiv1 (Table 2)intra-frame DPP only67.381.363.180.0
random (Table 2)none66.180.261.779.1
SMARTboth, in one kernel68.682.464.481.1

Relative to the age-only variant, SMART improves OVO-Avg/SBench by 0.8/1.9 on Qwen3-VL-8B and SBench by 2.5 on Qwen2.5-VL-7B. On Qwen2.5-VL-7B OVO-Avg, the age-only variant is 0.6 points higher, and the difference comes from the HLD task: excluding HLD, SMART is 1.2 points higher. Of all ablations in Table 2 and in this response (excluding random scores), replacing sis_i by age alone causes the largest StreamingBench drop on both backbones. On StreamingBench, both components of the score contribute: cross-frame novelty (si≡1s_i\equiv1 → si=wis_i=w_i) adds 0.9 on Qwen3-VL-8B and 1.8 on Qwen2.5-VL-7B, and the intra-frame DPP (si=wis_i=w_i → SMART) adds a further 1.0 and 0.7. On OVO-Avg, the two components are complementary: their combination in one kernel exceeds either component alone on both backbones (68.6 vs. 67.6 and 67.3; 64.4 vs. 63.7 and 63.1). We will add these variants to Table 2 in the revised paper.

W3, Q2 & Q3: Accuracy and latency of full rescoring.

We added the frozen-at-insertion variant (Q2, W3) and the current-event-only variant (W3):

Table R2. Rescoring variants (only the rescoring rule differs).

Rescoring ruleQwen3 OVO-AvgQwen3 OVO-BTQwen3 SBenchQwen2.5 OVO-AvgQwen2.5 SBench
Frozen at insertion67.757.481.563.780.3
Current event only68.157.781.964.180.7
Full memory (SMART)68.658.682.464.481.1

Relative to frozen scores, dynamic rescoring adds 0.9/0.9 OVO-Avg/SBench on Qwen3-VL-8B and 0.7/0.8 on Qwen2.5-VL-7B.

Full rescoring matters for two reasons. First, frozen scores persistently over-credit the first frame of each event relative to later redundant content. When that frame enters, Cf=∅C_f=\emptyset and w=1w=1, and this score is never revised, whereas tokens of later frames that repeat earlier content enter with references and receive low ww. The first frame thus keeps an inflated score after its content has been repeated, and only the age discount lowers its priority. Second, rescoring only the current event misses score recovery in finished events. After the older of two mutually redundant copies is evicted, the remaining copy should regain a high ww; without rescoring it keeps its low score and may be evicted next, leaving no copy of that content. Both effects concern older content, and consistent with this, the gap is largest on OVO-BT (1.2 points relative to frozen scores and 0.9 relative to current-event-only rescoring).

The latency in Table 4 is mainly an engineering issue rather than an inherent cost of the method: most of the 728 ms is implementation overhead. Our original implementation synchronized the CPU and GPU at every token selected by the greedy ordering and rescored every frame from scratch. We applied a few simple engineering optimizations that leave the outputs unchanged, such as batching the greedy ordering across frames, caching scores per event and recomputing only the events affected by an insertion or eviction, and replacing the full sort with a top-kk selection that keeps the original tie-breaking rule. With Qwen3-VL-8B at the default budget, memory management drops from 739 ms (Table 4) to 158 ms per frame, and the event boundaries and retained tokens are identical to those of the original implementation at every tested frame, so accuracy is unchanged. Further engineering, such as fusing the greedy ordering into a single GPU kernel, leaves room for additional speedups.

For the matched latency comparison in Q3, we measured the memory-update latency per frame of each variant with Qwen3-VL-8B on a single RTX 4090 at the default budget (mean / P95):

VariantMemory update (ms/frame)
si≡1s_i\equiv14 / 7
si=wis_i=w_i11 / 15
Frozen at insertion119 / 146
SMART, original implementation739 (Table 4)
SMART, optimized implementation158 / 185

The end-to-end query latency (input construction, generation, and text decoding) of SMART at the default budget is 288 ms. All variants share this query pipeline and the bound of BB visual tokens, so the scoring rule affects query latency only through the number of retained tokens and the length of the generated answer. Most of the update cost comes from scoring the newly inserted frame, which frozen scores also pay; with the optimized implementation, full rescoring adds 39 ms per frame on top of frozen scores (158 vs. 119 ms) and improves OVO-Avg by 0.9 (1.2 on OVO-BT), and visual encoding plus memory management runs at a real-time factor of 0.24 at 1 fps. We will add these variants, the latency results, and the optimized implementation to the revision.

Reviewer_gvnoOpenReview ↗

Response to Authors' Rebuttal

Thank you for the substantial effort in addressing my comments. The additional deterministic scoring controls resolve my second concern about the original ablation. They provide a much clearer attribution of the contributions from cross-frame novelty and intra-frame DPP, while also showing that the complete method does not consistently outperform age-only retention in every setting.

Regarding my first concern, the distinction between frame-level event segmentation and token-level redundancy is helpful. I agree that a missed event boundary does not necessarily suppress a local visual change. However, the remaining question concerns separate occurrences that are nearly identical in appearance within the same scene. Such occurrences can still carry distinct information for counting and temporal-order questions. The aggregate counting results are encouraging, but they do not directly establish whether this particular failure mode is handled. The promised examples would be useful if they show whether evidence of both occurrences survives in memory and supports the correct answer.

The new rescoring ablations and implementation optimizations also substantially address my third concern. They demonstrate measurable benefits from dynamic rescoring and considerably reduce its reported cost. Some uncertainty remains about the accuracy–latency trade-off: the latency of the current-event-only variant is not reported, although its accuracy is relatively close to full rescoring, and the simpler retention strategies remain substantially faster. Reporting the current-event-only latency would make this comparison more complete.

Reviewer_HtBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMART, a training-free, query-independent token-memory policy for streaming video understanding. SMART segments the stream into events, estimates each incoming token's redundancy against the current memory, uses an intra-frame determinantal point process (DPP) to assign diversity-aware scores, rescales scores by age, protects the most recent frames, and evicts low-scoring tokens under a fixed capacity. The method is evaluated with Qwen2.5-VL-7B and Qwen3-VL-8B on OVO-Bench and StreamingBench. The reported results and ablations suggest that dynamic rescoring, cross-frame deduplication, and marginal-contribution scoring improve accuracy under a four-frame-equivalent memory budget.

Critical strengths
  1. SMART addresses the practically important delayed-query setting with a causal, query-independent, fixed-capacity memory. The event-aware deduplication, recency discount, and recent-frame protection form a coherent policy rather than a collection of unrelated heuristics.

  2. The paper evaluates two current backbones and provides informative component, capacity, and latency analyses. In particular, explicitly reporting that memory management costs 739 ms per frame is useful and makes the systems trade-off auditable.

Critical weaknesses
  1. NovaCov ("Think in Sets for Streaming Video Token Compression") already formulates causal streaming compression as set selection by marginal contribution, uses a bounded recency-weighted historical reference, and greedily preserves content not covered by history. SMART replaces facility-location coverage with an intra-frame DPP plus a cross-frame scalar weight, but the central motivation and system abstraction are very close, and NovaCov is neither cited nor compared.
  2. SMART's DPP kernel is block-diagonal across frames, so cross-frame interaction is not part of the DPP objective itself; it is introduced only through the nearest-token heuristic weight. A matched comparison and a clearer statement of the distinct technical contribution are necessary.
  3. LiveStar (NeurIPS 2025) is also relevant: its Peak-End memory compression uses model-derived semantic importance, recency, and streaming KV-cache updates to retain informative content in long online streams, which targets proactive generation rather than delayed-query token memory. This makes it a useful novelty check for which parts of SMART's importance scoring, recent-frame protection, and online memory maintenance are genuinely new.
  4. On a 4090, memory management takes 739 ms per frame versus 58 ms for visual encoding, with 728 ms spent greedily rescoring every surviving frame whenever a new frame arrives. This is about 12.7 times the encoder cost and nearly exhausts a one-second budget even at the paper's unusually sparse 1-fps input, before accounting for the remaining online pipeline. The paper reports neither sustained end-to-end ingestion throughput nor scaling with realistic frame rates, concurrent queries, or larger memory budgets. Thus, the method is capacity-bounded but not yet shown to be computationally viable as a real-time memory policy.
Questions for authors
  1. How does SMART differ substantively from NovaCov beyond using DPP conditional variance rather than facility-location marginal gain?

  2. Since the DPP kernel is block-diagonal by frame, what justifies describing the final score as a joint set-wise marginal contribution relative to the current cross-frame memory? How much is gained over a simpler nearest-history novelty score plus within-frame diversity baseline?

  3. Can the method sustain ingestion at 1 fps and at conventional video rates when visual encoding, event segmentation, rescoring, eviction, cache updates, and query handling are measured end to end? Please report mean/P95 update latency and real-time factor over long streams.

5 comments

Thank you for the detailed comments. We address each point below.

W1 & Q1: Relation to NovaCov.

Thank you for pointing us to NovaCov. We agree that it is related, and we will cite and discuss it in Section 2.

Both methods view streaming compression as set selection, but beyond the choice of DPP conditional variance versus facility-location gain (Q1), they differ in three respects. First, NovaCov selects, once at admission, a fixed number of tokens per frame to pass to the LLM; the selected tokens are appended to a KV cache that grows with the stream, and a token's status is never revised after admission. SMART instead maintains an LLM-visible memory of fixed capacity, rescores all retained tokens as the memory changes (Section 3.3), and repeatedly decides which tokens of any age to evict, so tokens of all ages compete in a single eviction pool. Second, NovaCov compares new tokens with a separate prototype bank that the LLM does not see, whereas SMART measures redundancy against the tokens actually retained in memory, restricted to the same event, so similar content from different events is not treated as redundant. Third, recency plays a different role. In NovaCov, it weights the reference prototypes, i.e., it decides what counts as already seen; in SMART, it discounts the retention score of LLM-visible tokens and is embedded in the DPP kernel (Proposition 1), so eviction is greedy ordering on a single age-augmented kernel.

As for an empirical comparison, NovaCov (arXiv 2608.01169) first appeared on arXiv on August 2, 2026, after the July 17, 2026 cutoff for contemporaneous work in the ICLR 2027 reviewer guidelines, which do not require comparison with such work, and at the time of writing no code for NovaCov is publicly available. We therefore compare with NovaCov conceptually, as above, rather than empirically.

W2 & Q2: The block-diagonal kernel and the gain over a simpler baseline.

Section 3.3 states that sis_i approximates the conditional variance of token ii given the entire memory by two terms (end of the paragraph before "Non-Frozen Rescoring"). Each term has an exact conditional-variance interpretation under the DPP kernel: the cross-frame term wiw_i is the Schur complement of token ii conditioned on its best-matching same-event token (Appendix B), and the intra-frame term is its conditional variance given the same-frame tokens selected before it. Their combination is not the exact conditional variance given the whole cross-frame memory, which is why Section 3.3 calls sis_i an approximation. The block-diagonal kernel is the result of this factorization: cross-frame interaction enters through wiw_i, which conditions on the most redundant cross-frame token.

We do not use a joint kernel over the whole memory because its exact conditional variance saturates. It is the residual of each token after projection onto the span of all same-event tokens. As frames of an event accumulate, this span covers most of the feature space and the residual of every old token tends to zero, so the score loses its ability to rank tokens (Section 3.3, Appendix B). Conditioning on the best match does not saturate: a token without a near-duplicate keeps a high score however long the event becomes, and the cost drops from O(∣Cf∣2d+∣Cf∣3)O(|C_f|^2 d+|C_f|^3) to O(∣Cf∣ d)O(|C_f|\,d).

For the matched comparison, we keep the eviction policy fixed and change only sis_i:

sis_iQwen3 OVO-AvgQwen3 SBenchQwen2.5 OVO-AvgQwen2.5 SBench
si=wis_i=w_i (nearest-history novelty only)67.681.463.780.4
w≡1w\equiv1 (within-frame diversity only)67.381.363.180.0
wi+diw_i+d_i (novelty + diversity, computed separately)66.478.862.875.6
SMART (coupled in one kernel)68.682.464.481.1

The third row is the simpler baseline suggested in Q2: did_i is the intra-frame DPP score of the w≡1w\equiv1 variant, and the two scores are added with equal weights; both lie in [0,1][0,1] for unit-normalized features. Under SMART's kernel L~f=diag(w) GfGf⊤diag(w)\tilde L_f=\mathrm{diag}(\sqrt w)\,G_fG_f^\top\mathrm{diag}(\sqrt w), the conditional variance of token ii given a selected set CC equals wiw_i times its unweighted residual given the same CC. Novelty therefore gates diversity: a near-duplicate of a retained token (wi≈0w_i\approx0) scores close to zero however diverse it is within its frame, and the greedy ordering measures each token's diversity relative to the novel tokens selected before it. The additive baseline scores the two properties independently, so a near-duplicate that is diverse within its frame can still outrank a novel token. Empirically, SMART is 2.2/3.6 points higher than this equally weighted additive baseline on Qwen3-VL-8B and 1.6/5.5 on Qwen2.5-VL-7B, and the additive baseline falls below either component alone on all four metrics.

W3: Relation to LiveStar.

Thank you for the reference. LiveStar targets a different problem: it fine-tunes a Video-LLM for proactive response generation, and its Peak-End memory compression operates on frames, dropping frames outside a recent window with a probability determined by their relative caption perplexity and elapsed time.

Of the three parts listed, SMART differs from LiveStar in importance scoring and online memory maintenance, while recent-frame protection is shared with prior work. LiveStar scores each frame by its caption perplexity relative to its clip, which does not measure whether the frame's content is already covered by the retained memory. SMART is training-free, scores individual tokens, and the score is set-dependent: a token's value is its marginal contribution given the tokens currently retained in the same event, and it changes as the memory changes. For memory maintenance, SMART keeps a fixed token budget in which tokens of all ages compete in a single eviction pool, embeds the age discount in the DPP kernel (Proposition 1), and rescores all retained tokens at every frame, whereas LiveStar applies a fixed recency window and probabilistic dropping to frames. Keeping the most recent frame intact is shared with guard-band methods, as Section 2 states, and we do not claim it as a contribution. We will add LiveStar to Section 2 with this distinction.

W4 & Q3: Computational cost and sustained end-to-end ingestion.

The latency in Table 4 is mainly an engineering issue rather than an inherent cost of the method: most of the 739 ms is implementation overhead. Our original implementation synchronized the CPU and GPU at every token selected by the greedy ordering and rescored every frame from scratch. We applied a few simple engineering optimizations that leave the outputs unchanged, such as batching the greedy ordering across frames, caching scores per event and recomputing only the events affected by an insertion or eviction, and replacing the full sort with a top-kk selection. Memory management drops from 739 ms (Table 4) to 158 ms per frame, with identical event boundaries and retained tokens at every tested frame. Further engineering, such as fusing the greedy ordering into a single GPU kernel, leaves room for additional speedups.

Table R3 compares both implementations with Qwen3-VL-8B on the same consumer RTX 4090 at the default budget; per-frame latency covers visual encoding, event segmentation, rescoring, eviction, and cache updates:

SettingOriginal implementation (Table 4)Optimized implementation
Visual encoding + memory management, mean / P95 (ms/frame)797 / –238 / 326
Real-time factor at 1 / 2 / 4 fps0.80 / 1.59 / 3.190.24 / 0.48 / 0.95
Estimated throughput from mean latency (fps)1.254.2

At 4 fps the mean latency fits within the 250 ms frame interval. The latency also stays stable over long streams and the tested budgets. Per-step computation is bounded by the memory capacity and does not depend on the stream length (Section 3.5), and window-averaged per-frame latency stays within ±1.1% of the overall mean over a 1,200-frame stream (300-frame windows). Over the tested budgets, memory management, timed separately on cached features, remains nearly constant:

B/nB/n248
Memory management, mean / P95 (ms/frame)159 / 197158 / 185160 / 190

Our accuracy results use 1 fps, which is a standard setting for streaming Video-LLMs rather than an unusually sparse one. TimeChat-Online (Yao et al., 2025) regards 1–10 fps as high frame rates for real-world streams. The compared methods report results at 1 fps, including TimeChat-Online, StreamForest (Zeng et al., 2025), FluxMem (Xie et al., 2026), Dispider (Qian et al., 2025), WeaveTime (Zhang et al., 2026), and HERMES (Zhang et al., 2026), with VideoLLM-online (Chen et al., 2024) at 2 fps. Several training-free memory methods use 0.5 fps or lower, such as StreamingTOM (Chen et al., 2026), InfiniPot-V (Kim et al., 2025), and HERMES (on LLaVA-OneVision). We use 1 fps so that our results are directly comparable to the published numbers, following the principle of OVO-Bench (Li et al., 2025) of keeping the frame rate the same across models.

For conventional video rates of 24–30 fps, SMART, like all compared streaming Video-LLMs, subsamples the source stream before visual encoding, so ingesting such a camera stream amounts to decoding it and sampling frames at the chosen rate. Processing every frame would leave 33–42 ms per frame, below the 60 ms that the backbone's visual encoder alone takes per frame at its default resolution in this measurement; this bound applies to any method built on this backbone. Sampling rates above 1 fps remain possible: the mean per-frame latency corresponds to an estimated throughput of 4.2 fps (Table R3).

For concurrent queries, the memory is query-independent, so all queries read the same memory snapshot and no memory has to be maintained per query. Each query costs one LLM call over the retained visual tokens, 288 ms at the default budget (415 and 556 ms at B/nB/n = 6 and 8), including input construction, generation, and text decoding. Four queries take 324 ms when batched, compared with 1,156 ms when run one after another.

Reviewer_bdk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies bounded visual token memory management in streaming video understanding. Under the constraints that the timing and content of future queries are unknown and evicted tokens cannot be recovered, the authors propose SMART, a training-free, query-independent memory management method. SMART restricts the scope of cross-frame redundancy through adaptive event segmentation and combines intra-event cross-frame nearest-match weights with intra-frame determinantal point processes (DPPs) to approximate each token's marginal contribution to the current memory. Upon the arrival of each new frame, the system recomputes scores for all tokens currently in memory, applies exponential time discounting for eviction, and fully preserves the most recent frame. The authors evaluate SMART on the Real-Time Visual Perception and Backward Tracing tasks of OVO-Bench and the Real-Time Visual Understanding subset of StreamingBench, using Qwen2.5-VL-7B and Qwen3-VL-8B backbones. With Qwen3-VL-8B, SMART achieves 68.6% OVO-Avg and 82.4% accuracy on StreamingBench. Ablation studies show that removing cross-frame deduplication, marginal contribution scoring, time discounting, or recent-frame protection lowers the aggregate metrics on both benchmarks.

Critical strengths
  1. Dynamic redundancy modeling accounts for temporal context and current memory Equations (3) to (7) couple intra-event cross-frame coverage with intra-frame token complementarity. Scores for surviving tokens are recomputed whenever a new frame arrives. This allows retention values to evolve as memory changes and directly addresses the streaming setting, in which memory contents continually change and eviction is irreversible. This is the paper's most valuable methodological contribution.
  2. Controlled ablations support the effectiveness of individual modules Table 2 shows that, on both backbones, removing cross-frame deduplication reduces OVO-Avg by 1.3 percentage points and StreamingBench accuracy by 1.1 percentage points. Removing time discounting or recent-frame protection also lowers both aggregate metrics. These controlled comparisons support the contribution of individual components to the final performance beyond differences between backbones.
Critical weaknesses
  1. The central claim is that marginal contribution is a better retention criterion than individual saliency. Table 1 compares methods using different backbones and training data, making it difficult to attribute the performance gaps solely to the memory-management strategy. Table 2 includes randomized scoring and a DPP variant without cross-frame weights, but no attention- or value-norm scoring under the same memory policy. These ablations support useful design choices but do not directly establish the claimed advantage over saliency. A matched comparison is important for assessing this contribution.
  2. Table 2 and Section 4.3 specify C_f=M. Taken literally in Equation (5), this includes self-matches: unit-normalized features would give w_i=0 and a zero DPP kernel. This may simply omit the exclusion of same-frame tokens. The implementation needs clarification before this ablation can reliably support the claim that restricting cross-frame redundancy comparisons to the same event is beneficial; a notation correction would resolve this concern if same-frame tokens were excluded.
  3. Table 4 reports 739 ms per frame for memory management on an RTX 4090, including 728 ms for greedy ordering, compared with 58 ms for visual encoding. The authors acknowledge this bottleneck. Because the evaluation samples frames at 1 fps, it remains unclear whether SMART can sustain higher sampling rates or scale efficiently to larger memory budgets. Throughput and latency measurements across sampling rates and budgets, including query-time inference, would clarify the accuracy-efficiency trade-off and practical applicability.
Questions for authors
  1. What exact candidate set was used in the global deduplication experiment in Table 2? Were all tokens from the current frame excluded? If so, please correct the notation. If self-matching was included, please explain how weight collapse was avoided and whether it affected the reported results.
  2. Can you provide a comparison using the same backbone, token budget, time discounting, and protection rules, replacing the proposed marginal-contribution score s_i with value-vector norms or attention scores? Consistently outperforming these saliency-based alternatives would provide stronger evidence for the proposed marginal-contribution criterion.Please also report the memory-update latency of each scoring variant under the same backbone and token budget.
  3. Appendix C derives a reference initialization of h_0≈0.415 for the default capacity B/n=4, whereas the experiments use h=0.3. How was the final value selected relative to this reference, and what data were used for hyperparameter selection? Were the OVO-Bench sensitivity analyses used to tune the parameters? If so, was a separate validation split used to avoid tuning on the evaluation data?
  4. Because the newly arrived frame is marked as protected before eviction, while the previously protected P frames are not unmarked until after the eviction loop, the algorithm can temporarily have P+1 protected frames during eviction when memory overflows (two frames when P=1). Does this reflect the implementation used in the experiments? Please clarify the actual update order and protection policy underlying the results in Tables 1 and 2.
4 comments

W1 & Q2: Matched comparison with saliency scores.

We added the matched comparison requested in Q2. Attention and value-norm scores replace sis_i under the same backbone, budget, age discount, and protection rule: attention is the mean attention a token receives in full-attention layers of the LLM backbone, and value norm is the mean ℓ2\ell_2 norm of the token's value vectors, following InfiniPot-V (Kim et al., 2025); both are query-independent.

sis_iQwen3 OVO-AvgQwen3 SBenchQwen2.5 OVO-AvgQwen2.5 SBenchMemory update (ms/frame)
attention67.380.963.578.930
value norm65.578.361.277.668
SMART68.682.464.481.1158

Latency is the mean memory-update time per frame, measured with Qwen3-VL-8B and the same budget on a single RTX 4090; for SMART it is that of the optimized implementation described in W3 below, which retains the same tokens as the original implementation (739 ms per frame in Table 4). SMART outperforms both saliency scores on both backbones, by 0.9–3.2 OVO-Avg and 1.5–4.1 SBench.

Table 1 also contains training-free methods on the same backbone: HERMES and FluxMem use Qwen2.5-VL-7B, as does our Qwen2.5 row, and SMART reaches 64.4/81.1 OVO-Avg/SBench versus 59.2/79.4 for HERMES and 55.9/78.1 for FluxMem. The HERMES paper also reports Qwen3-VL-8B at 2 fps: 61.3/81.3, versus 68.6/82.4 for SMART at 1 fps. The gains over trained methods on the same backbone are consistent with these matched comparisons.

W2 & Q1: Candidate set of the global-deduplication variant.

Thank you for catching this. It is a notation error. In the implementation, the candidate set is every token in memory except those of frame ff, Cf={j∈M:j∉f}C_f=\{j\in\mathcal{M}: j\notin f\}, i.e., Eq. (5) without the same-event condition. All tokens of the current frame are excluded, so there is no self-match and no weight collapse, and the reported numbers are valid. We will correct the notation in Table 2 and Section 4.3. The ablation therefore supports event scoping: removing the event restriction lowers OVO-Avg/SBench by 0.7/1.3 on Qwen3-VL-8B and 0.5/0.6 on Qwen2.5-VL-7B.

W3: Throughput and latency across sampling rates and budgets.

The latency in Table 4 is mainly an engineering issue rather than an inherent cost of the method: most of the 728 ms is implementation overhead. We applied a few simple engineering optimizations that leave the outputs unchanged, such as batching the greedy ordering across frames without per-token CPU–GPU synchronization, caching scores per event and recomputing only the events affected by an insertion or eviction, and replacing the full sort with a top-kk selection. With Qwen3-VL-8B at the default budget, memory management drops from 739 ms (Table 4) to 158 ms per frame, and the event boundaries and retained tokens are identical to those of the original implementation at every tested frame, so accuracy is unchanged. Further engineering, such as fusing the greedy ordering into a single GPU kernel, leaves room for additional speedups.

For sampling rates, the table below compares both implementations with Qwen3-VL-8B on the same RTX 4090, excluding video decoding. With the optimized implementation, the mean latency fits within the 250 ms frame interval even at 4 fps:

SettingOriginal implementation (Table 4)Optimized implementation
Visual encoding + memory management, mean / P95 (ms/frame)797 / –238 / 326
Real-time factor at 1 / 2 / 4 fps0.80 / 1.59 / 3.190.24 / 0.48 / 0.95
Estimated throughput from mean latency (fps)1.254.2

For budgets, memory management (timed separately on cached features) stays nearly constant over the tested budgets, and per-frame latency does not grow with the stream length:

B/nB/n248
Memory management, mean / P95 (ms/frame)159 / 197158 / 185160 / 190

Finally, query-time inference, including input construction, generation, and text decoding, takes 288, 415, and 556 ms per query at B/nB/n = 4, 6, and 8; it grows with the number of retained visual tokens, which is at most BB, and the scoring policy affects it only through that number and the length of the generated answer.

Q3: Selection of hh and validation data.

Thank you for pointing out that our description of the protocol was incomplete.

h0h_0 is an analytic starting point that requires no data: it makes the expected steady-state retention sum to the capacity (Appendix C) and is intended as the default for deployment, where no validation data exist. For the benchmarks, we selected h=0.3h=0.3, starting from h0h_0, on a small validation subset of 100 questions randomly sampled from the two benchmarks (50 from each). Hyperparameters were selected only on this subset, not on the full benchmarks, and the same values are used for both backbones and both benchmarks. Figure 3(a) shows that h0h_0 lies within the tested range h∈[0.08,0.64]h\in[0.08, 0.64], over which OVO-Avg varies by at most about 0.8 points and stays far above all compared methods, so using h0h_0 without any validation data would not change our conclusions. As for the sensitivity analyses on OVO-Bench, they were not used for tuning, and Figure 3 is a post-hoc analysis.

Q4: Update order and protected frames.

Thank you for the careful reading. The implementation unmarks the previously protected frame before eviction, so exactly one frame, the newly arrived one, is protected during eviction, as described in the text (P=1P=1); the pseudocode places the unmarking line at the wrong position. The actual order at each step is: (1) event segmentation; (2) insert the new frame, mark it as protected, and unmark older frames; (3) rescoring; (4) eviction among unprotected tokens. All results in Tables 1 and 2 follow this order. We will move the line before Step 4 in Algorithm 1.

Reviewer_bdk9OpenReview ↗

maintaining my original score

Thank you for the additional controlled experiments. Under matched conditions, SMART’s marginal-contribution scoring outperforms the attention and value-norm baselines on both backbones, addressing my concern about the fairness of the original comparison. The authors also clarified that the candidate set for global deduplication excludes all tokens from the frame being scored. With P=1, as used in the experiments, only the newly arrived frame is protected during eviction. I consider these concerns largely resolved.

My main remaining concern is latency in the streaming setting. The optimizations reduce SMART’s memory-update time from 739 to 158 ms per frame, a substantial improvement. On Qwen3-VL-8B, SMART improves on attention scoring by 1.3 percentage points on OVO-Avg and 1.5 points on SBench, but its memory updates take an additional 128 ms per frame. The current comparison holds the token budget fixed. It remains unclear whether SMART would retain its accuracy advantage if the two methods were matched on memory-update time per frame.

I recognize the accuracy gains shown by the additional experiments. For a streaming method, however, the extra latency is incurred on every frame, and that cost weighs more heavily in my assessment. I am therefore maintaining my original score.

Thank you for the follow-up. We understand the motivation for comparing the methods under a common update-time budget. For the two scoring rules as evaluated, token selection is determined by the input features, scoring rule, and token budget. Once an update is complete, additional time does not further refine the selection. With these conditions fixed, matching the allotted update time alone would therefore leave the retained tokens unchanged for either method.

To assess the recurring update cost in practice, we added a streaming run on one RTX 4090 over a 60s submission schedule (including source-video decoding) that submitted 1 frame and a batch of four concurrent queries reading the same memory snapshot every second (60 frames and 240 queries total). Over 60.647 s of total wall-clock runtime, all 60 frames and 240 queries completed, yielding 0.989 frames/s and 3.957 queries/s—roughly 1 frame/s and 4 queries/s end to end. This demonstrates that SMART's update cost is compatible with the tested 1 fps streaming workload while also serving four queries per second in batches.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and all reviewers for their constructive evaluations. Following two rounds of discussion, Reviewer gvno stated that the concerns from the initial review were largely addressed and raised the rating. Reviewer HtBY is also inclined to raise the score, while Reviewer bdk9 considers the comparison and implementation concerns largely resolved but maintains the original score due to latency. We summarize the strengths recognized across the reviews and our responses to the main concerns.

Strengths recognized across the reviews.

All three reviews found the paper informative, and Reviewers HtBY and bdk9 also noted its clear presentation. Methodologically, reviewers praised SMART's coherent, training-free, query-independent policy for delayed-query streaming (Reviewer HtBY), its event-scoped matching paired with intra-frame DPP diversity (Reviewer gvno), and dynamic score recomputation under irreversible eviction as a key contribution (Reviewer bdk9). Empirically, reviewers highlighted SMART's highest listed accuracy on OVO-Bench and StreamingBench (Reviewer gvno), controlled ablations supporting individual modules and event scoping (Reviewers bdk9, gvno), and auditable latency and capacity analyses across two backbones (Reviewer HtBY).

1. Scoring attribution and matched baselines (Reviewers gvno, bdk9, HtBY).

Reviewers pointed out that the original ablations used random scores rather than matched deterministic, additive, or saliency baselines. We added all requested variants under identical backbones, budgets, and retention rules, showing that SMART outperforms attention and value-norm saliency, each individual scoring component, and the additive baseline across both backbones and benchmarks. Reviewer gvno confirmed that these deterministic additions resolved the concern regarding ablation attribution.

2. Dynamic rescoring, latency, and scalability (Reviewers gvno, HtBY, bdk9).

All three reviews flagged the 739 ms latency in Table 4, questioning the cost–benefit trade-off of full rescoring and scalability across sampling rates, stream lengths, budgets, and concurrent queries. We isolated full rescoring against frozen and current-event-only variants, showing higher accuracy over both while incurring nearly the same latency as the latter. Through output-preserving engineering optimizations, we reduced memory-management latency from 739 ms to 158 ms and end-to-end ingestion to an average of 238 ms per frame (within the 250 ms interval at 4 fps), while verifying stable per-frame update latency across stream lengths and budgets, batched execution for concurrent queries, and alignment with standard 1 fps protocols. Reviewer gvno found that these rescoring ablations and optimizations substantially addressed the latency concern, and we supplied the requested current-event-only latency in round 2.

3. Segmentation and implementation clarifications (Reviewers gvno, bdk9).

Regarding Reviewer gvno's question on repeated actions, we clarified that redundancy is determined at the token level rather than by frame boundaries and committed to adding illustrative examples in the revision. Addressing Reviewer bdk9's questions on notation and protection order, we confirmed that the implementation excludes same-frame tokens and protects a single frame before eviction, requiring only notation and pseudocode updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.