- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies fixed-capacity visual-token memory for query-independent streaming video question answering. Its training-free policy segments frames into causal content events, uses same-event nearest-neighbor residuals to modulate per-frame DPP greedy scores, and applies age decay while preserving recent frames. Experiments with Qwen2.5-VL-7B and Qwen3-VL-8B report the highest listed aggregate scores on OVO-Bench and StreamingBench, together with component-level ablations.
Critical strengths
-
Event-scoped cross-frame matching is a useful way to distinguish repetition within one continuous event from similar content observed at different times. Combining this with intra-frame DPP scores gives the policy a concrete mechanism for preserving both temporal novelty and spatial complementarity without training.
-
SMART has the highest listed OVO-Avg and StreamingBench accuracy with both tested backbones in Table 1. In the fixed-backbone results in Table 2, removing cross-frame deduplication lowers OVO-Avg by 1.3 points and StreamingBench by 1.1 points on both backbones. Global deduplication also lowers all four aggregate results, which supports restricting redundancy to the detected event.
Critical weaknesses
-
SMART can distinguish redundancy from a repeated occurrence only when the event labels are correct, but Equations (1)–(2) detect a new event using the median token similarity of the whole frame. A small but important action in an otherwise static scene may not change this median, so two separate occurrences can remain in the same event and be treated as duplicates. The global-deduplication ablation in Table 2 shows that event scoping helps, but Figures 6–8 mainly show scene or camera changes and do not test repeated actions, counting, or temporal order within the same scene.
-
Table 2 does not support the claim that marginal-contribution scoring is the largest source of improvement. Replacing with random values turns eviction into a noisy policy, so its 2.0–2.7 point drop cannot be compared directly with the cleaner component removals. A uniform-score age-only control and an control without the intra-frame DPP are needed to isolate what the proposed score adds.
-
Full-memory rescoring is presented as a central contribution, but its accuracy benefit is never isolated even though it dominates the cost. Table 4 reports 728 ms for greedy ordering and 739 ms for memory management, compared with 58 ms for visual encoding, while Table 2 has no frozen-score or current-event-only rescoring control. Without those controls or latency measurements for simpler retention baselines, the paper does not show that full rescoring provides a better accuracy–latency trade-off.
Questions for authors
-
A few qualitative examples showing how Equations (1)–(2) segment repeated actions in the same scene would be helpful. Cases involving counting or temporal-order questions would clarify whether separate occurrences remain distinguishable in memory.
-
Adding , , and frozen-at-insertion score controls to Table 2 would make the ablation more informative. Under the same age discount and protection settings, these rows would separate the effects of cross-frame novelty, intra-frame DPP, and dynamic rescoring.
-
A matched latency comparison for SMART and the three controls above would also be useful. Reporting both per-frame update latency and end-to-end query latency would show whether full-memory rescoring is worth its 728 ms cost.
6 comments
W3, Q2 & Q3: Accuracy and latency of full rescoring.
We added the frozen-at-insertion variant (Q2, W3) and the current-event-only variant (W3):
Table R2. Rescoring variants (only the rescoring rule differs).
| Rescoring rule | Qwen3 OVO-Avg | Qwen3 OVO-BT | Qwen3 SBench | Qwen2.5 OVO-Avg | Qwen2.5 SBench |
|---|---|---|---|---|---|
| Frozen at insertion | 67.7 | 57.4 | 81.5 | 63.7 | 80.3 |
| Current event only | 68.1 | 57.7 | 81.9 | 64.1 | 80.7 |
| Full memory (SMART) | 68.6 | 58.6 | 82.4 | 64.4 | 81.1 |
Relative to frozen scores, dynamic rescoring adds 0.9/0.9 OVO-Avg/SBench on Qwen3-VL-8B and 0.7/0.8 on Qwen2.5-VL-7B.
Full rescoring matters for two reasons. First, frozen scores persistently over-credit the first frame of each event relative to later redundant content. When that frame enters, and , and this score is never revised, whereas tokens of later frames that repeat earlier content enter with references and receive low . The first frame thus keeps an inflated score after its content has been repeated, and only the age discount lowers its priority. Second, rescoring only the current event misses score recovery in finished events. After the older of two mutually redundant copies is evicted, the remaining copy should regain a high ; without rescoring it keeps its low score and may be evicted next, leaving no copy of that content. Both effects concern older content, and consistent with this, the gap is largest on OVO-BT (1.2 points relative to frozen scores and 0.9 relative to current-event-only rescoring).
The latency in Table 4 is mainly an engineering issue rather than an inherent cost of the method: most of the 728 ms is implementation overhead. Our original implementation synchronized the CPU and GPU at every token selected by the greedy ordering and rescored every frame from scratch. We applied a few simple engineering optimizations that leave the outputs unchanged, such as batching the greedy ordering across frames, caching scores per event and recomputing only the events affected by an insertion or eviction, and replacing the full sort with a top- selection that keeps the original tie-breaking rule. With Qwen3-VL-8B at the default budget, memory management drops from 739 ms (Table 4) to 158 ms per frame, and the event boundaries and retained tokens are identical to those of the original implementation at every tested frame, so accuracy is unchanged. Further engineering, such as fusing the greedy ordering into a single GPU kernel, leaves room for additional speedups.
For the matched latency comparison in Q3, we measured the memory-update latency per frame of each variant with Qwen3-VL-8B on a single RTX 4090 at the default budget (mean / P95):
| Variant | Memory update (ms/frame) |
|---|---|
| 4 / 7 | |
| 11 / 15 | |
| Frozen at insertion | 119 / 146 |
| SMART, original implementation | 739 (Table 4) |
| SMART, optimized implementation | 158 / 185 |
The end-to-end query latency (input construction, generation, and text decoding) of SMART at the default budget is 288 ms. All variants share this query pipeline and the bound of visual tokens, so the scoring rule affects query latency only through the number of retained tokens and the length of the generated answer. Most of the update cost comes from scoring the newly inserted frame, which frozen scores also pay; with the optimized implementation, full rescoring adds 39 ms per frame on top of frozen scores (158 vs. 119 ms) and improves OVO-Avg by 0.9 (1.2 on OVO-BT), and visual encoding plus memory management runs at a real-time factor of 0.24 at 1 fps. We will add these variants, the latency results, and the optimized implementation to the revision.
Response to Authors' Rebuttal
Thank you for the substantial effort in addressing my comments. The additional deterministic scoring controls resolve my second concern about the original ablation. They provide a much clearer attribution of the contributions from cross-frame novelty and intra-frame DPP, while also showing that the complete method does not consistently outperform age-only retention in every setting.
Regarding my first concern, the distinction between frame-level event segmentation and token-level redundancy is helpful. I agree that a missed event boundary does not necessarily suppress a local visual change. However, the remaining question concerns separate occurrences that are nearly identical in appearance within the same scene. Such occurrences can still carry distinct information for counting and temporal-order questions. The aggregate counting results are encouraging, but they do not directly establish whether this particular failure mode is handled. The promised examples would be useful if they show whether evidence of both occurrences survives in memory and supports the correct answer.
The new rescoring ablations and implementation optimizations also substantially address my third concern. They demonstrate measurable benefits from dynamic rescoring and considerably reduce its reported cost. Some uncertainty remains about the accuracy–latency trade-off: the latency of the current-event-only variant is not reported, although its accuracy is relatively close to full rescoring, and the simpler retention strategies remain substantially faster. Reporting the current-event-only latency would make this comparison more complete.
Thank you for the constructive comments. We address each point below.
W1 & Q1: Separate occurrences within the same scene.
Two occurrences in the same event are not treated as duplicates merely because they share the event label, since SMART decides the scope of comparison and redundancy at two different levels. Event segmentation (Eqs. 1–2) operates on frames and only decides which retained frames serve as references, whereas redundancy is decided per token by its best match within that scope, wi=1−(maxj∈Cffi⊤fj)2 (Eq. 5), not by the frame-level median.
A missed boundary therefore does not necessarily suppress a local change: tokens with low similarity to the retained references, such as those on a moving hand or a changed object, can still receive high novelty weights, while the unchanged background receives a low wi, as intended. Global deduplication in Table 2 is the extreme case with no boundaries at all, and it lowers OVO-Avg/SBench by 0.7/1.3 (Qwen3-VL-8B) and 0.5/0.6 (Qwen2.5-VL-7B). Likewise, at θ=0.55, where 90% of the windows contain a single event (Figure 4), OVO-Avg changes by about 0.2 points (Figure 3(b)). Overall accuracy therefore tolerates segmentation errors well.
For counting, which Q1 raises, the StreamingBench counting subtask (CT) covers both object and action counting. On both backbones, SMART has higher overall CT accuracy than both the age-only variant and the variant without cross-frame deduplication:
The remaining case is a repetition that is nearly identical at the token level (same position, pose, and appearance) to a retained occurrence in the same event. The visual evidence itself is then repeated. We will add qualitative examples of repeated actions within the same scene, including counting and temporal-order cases, to the revised paper.
W2 & Q2: Variants of the scoring term.
The randomized variant keeps the age discount and the protection rule and replaces only the content score si. To attribute the gain without relying on random scores, we add the deterministic si≡1 and si=wi variants requested in Q2, which also change only si (the frozen-score variant is reported in the next part):
Table R1. Scoring variants (only si differs).
Relative to the age-only variant, SMART improves OVO-Avg/SBench by 0.8/1.9 on Qwen3-VL-8B and SBench by 2.5 on Qwen2.5-VL-7B. On Qwen2.5-VL-7B OVO-Avg, the age-only variant is 0.6 points higher, and the difference comes from the HLD task: excluding HLD, SMART is 1.2 points higher. Of all ablations in Table 2 and in this response (excluding random scores), replacing si by age alone causes the largest StreamingBench drop on both backbones. On StreamingBench, both components of the score contribute: cross-frame novelty (si≡1 → si=wi) adds 0.9 on Qwen3-VL-8B and 1.8 on Qwen2.5-VL-7B, and the intra-frame DPP (si=wi → SMART) adds a further 1.0 and 0.7. On OVO-Avg, the two components are complementary: their combination in one kernel exceeds either component alone on both backbones (68.6 vs. 67.6 and 67.3; 64.4 vs. 63.7 and 63.1). We will add these variants to Table 2 in the revised paper.