- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
StreamIndex separates stream-time indexing from query-time visual encoding for streaming video understanding. It uses histogram-based clustering, query-guided retrieval, and motion-based feature reuse to reduce computation. Experiments on Qwen-VL and LLaVA-OneVision show improved multi-stream serving capacity with competitive, backbone-dependent accuracy on OVO-Bench and StreamingBench.
Critical strengths
- The motivation of the paper is clear. When multiple video streams share a GPU, continuous visual encoding is a significant bottleneck. The query load experiment indicates that StreamIndex supports a higher workload than the comparison baselines.
- The proposed method framework is training-free and highly practical for real-world applications and deployments.
Critical weaknesses
- The novelty of the paper is slightly limited, mainly based on the combination of existing methods. And the paper does not compare histogram-based clustering with fixed-length grouping or periodic keyframe selection under similar encoding budgets. It is therefore unclear whether the proposed clustering is better than simply encoding fewer frames.
Questions for authors
See Critical Weaknesses and:
- It would be beneficial to provide some qualitative examples of successes and failures, illustrating the scenarios in which StreamIndex successfully preserves key visual evidence versus those in which such evidence is missed. (e.g. How reliable is cluster retrieval when semantically important events involve only subtle visual changes?)
- The paper does not clearly describe how historical video frames are stored and accessed during query-time processing. It would be helpful if the authors could clarify the storage mechanism and how the associated data-loading overhead is accounted for in the latency evaluation.
4 comments
Dear Reviewer F4Wb,
Following up on our rebuttal, we wanted to kindly check whether you might have any further concerns or feedback to share.
Your suggestions have helped us improve the presentation and evaluation of our work. In our response, we have further clarified the contribution of our framework, provided additional clustering comparisons and qualitative success and failure examples, and explained historical-frame storage and how frame-access and transfer costs are included in our latency measurements.
We greatly appreciate your time and consideration. Please feel free to share any thoughts you may have; we would be happy to provide further clarification if helpful.
Best regards,
The Authors
Thank you for the author's explanation, which has resolved some of my doubts. I will raise my score, but I may not be confident.
Dear Reviewer F4Wb,
Thank you very much for your positive feedback and willingness to raise your rating. We sincerely appreciate your recognition of our work and your constructive suggestions. In the final version, we will carefully incorporate your comments and make further revisions to clarify the novelty and technical contributions of StreamIndex, strengthen the completeness of our experimental analysis.
Your feedback has been invaluable in helping us improve the quality and clarity of our work. Thank you again for your encouragement and support!
Best regards, The Authors
Response to Reviewer F4Wb
Thank you for your careful reading and constructive suggestions. Your recognition of our clear motivation, higher supported workload, and practical, training-free design is a great encouragement to us!
Following your suggestions, we add clustering comparisons and qualitative examples, and clarify our contribution and frame-access costs.
W1 (Part 1). Core contribution
StreamIndex's core contribution is identifying and addressing two scalability bottlenecks in existing streaming video methods, as illustrated in
Figure 1 (right): continuous visual encoding can exceed the available compute budget, while accumulating encoded history can exhaust GPU memory. Both arise from encoding frames before knowing their relevance to a query.StreamIndex uses one representative index that supports both retrieval and reuse: representative embeddings identify query-relevant clusters, and cached representative features support selective feature reuse during query-time encoding. This unified design decouples stream-time indexing from query-time visual encoding to balance answer accuracy and computation cost, thus supporting multi-stream scalability.
W1 (Part 2). Clustering ablation
To evaluate cluster segmentation, we compare histogram clustering with fixed-length strategies (Fixed-4, Fixed-6, and Fixed-8), which group every 4, 6, and 8 consecutive sampled frames into one cluster, respectively. The comparison on OVO-Bench is included in
Table 6of the revised manuscript.Finer fixed-length segmentation improves accuracy but increases encoding cost, consistent with the expected trade-off: from Fixed-8 to Fixed-4, Total accuracy rises from 52.00% to 52.84%, while Total Encoded Frames increases from 31.88 to 57.80. StreamIndex's dynamic histogram-based segmentation balances accuracy and total computation cost.
Q1. Success and failure cases
We have added qualitative examples in
Figures 8–9.Figure 8(a,d)shows successful evidence preservation and correct color recognition; (c) answers correctly with partial evidence. In contrast, (b) misses the clusters containing lamp-state evidence, while (e) retrieves the relevant cluster but selects frames after the target disappears. Thus, subtle state evidence can be missed at retrieval, and successful cluster retrieval does not ensure evidence retention.Figure 9compares three frame-selection rules: adjacent-frame differences (StreamIndex default), first-to-current differences, and cumulative adjacent-frame differences. No rule consistently preserves the most evidence across all three cases: adjacent-frame differences retain more evidence in (C), while cumulative differences retain more in (B) but miss it in (A) and (C).Since relevant evidence can occur anywhere within a cluster, no fixed, query-agnostic frame-selection heuristic can guarantee evidence retention across all scenarios. StreamIndex offers a practical heuristic with a failure boundary.
Q2. Frame storage and access cost
StreamIndex stores decoded frame pixels in CPU memory and cluster representative embeddings in GPU memory.
Frame access and CPU-to-GPU transfer are included in our end-to-end query latency, which covers retrieval, query-time encoding, and generation. In an OVO-Bench measurement, historical-frame access averaged 0.0097 ms, accounting for 0.0011% of query latency. Transferring historical cluster frames from CPU to GPU accounts for only 0.28% of the full end-to-end latency, indicating a small transfer overhead.
Summary
In summary, thank you again for helping us clarify our contribution, strengthen the clustering evaluation, illustrate successes and failures, and explain storage and latency accounting.
We hope our responses have addressed your concerns and would sincerely appreciate further feedback and your reconsideration of the rating in light of the additional evidence.