acceptodds
Under review as a conference paper at ICLR 2027

StreamIndex: Decoupling Indexing from Encoding for Scalable Streaming Video Understanding

Abstract

Serving concurrent video streams under a fixed GPU budget requires sustained ingestion and timely responses to unpredictable queries. Existing streaming video MLLMs typically encode incoming frames before deciding which information to retain or retrieve, making continuous visual computation a scalability bottleneck. We propose StreamIndex, a training-free framework that decouples stream-time indexing from query-time visual encoding. During ingestion, lightweight CPU-side statistics group similar frames into clusters, and only one representative per finalized cluster is encoded for retrieval. When a query arrives, the system retrieves relevant clusters and recovers details through motion-based frame selection and selective reuse of stable spatial features. This design reduces continuous encoding while controlling the additional work on the query path. Experiments on the Qwen-VL and LLaVA-OneVision families show improved multi-stream scalability with competitive accuracy on OVO-Bench and StreamingBench. In a 16-stream 1080p query-load test at two queries per minute per stream, StreamIndex achieves a P95 response latency of 5.57 seconds, while existing state-of-the-art baselines either violate the real-time bound or run out of memory. The results expose a practical trade-off between ingestion cost, query-time computation, and answer quality. The code is available at https://anonymous.4open.science/r/StreamIndex-7D6E.

Primary area:
generative models
Keywords:
Streaming Video Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F4WbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

StreamIndex separates stream-time indexing from query-time visual encoding for streaming video understanding. It uses histogram-based clustering, query-guided retrieval, and motion-based feature reuse to reduce computation. Experiments on Qwen-VL and LLaVA-OneVision show improved multi-stream serving capacity with competitive, backbone-dependent accuracy on OVO-Bench and StreamingBench.

Critical strengths
  1. The motivation of the paper is clear. When multiple video streams share a GPU, continuous visual encoding is a significant bottleneck. The query load experiment indicates that StreamIndex supports a higher workload than the comparison baselines.
  2. The proposed method framework is training-free and highly practical for real-world applications and deployments.
Critical weaknesses
  1. The novelty of the paper is slightly limited, mainly based on the combination of existing methods. And the paper does not compare histogram-based clustering with fixed-length grouping or periodic keyframe selection under similar encoding budgets. It is therefore unclear whether the proposed clustering is better than simply encoding fewer frames.
Questions for authors

See Critical Weaknesses and:

  1. It would be beneficial to provide some qualitative examples of successes and failures, illustrating the scenarios in which StreamIndex successfully preserves key visual evidence versus those in which such evidence is missed. (e.g. How reliable is cluster retrieval when semantically important events involve only subtle visual changes?)
  2. The paper does not clearly describe how historical video frames are stored and accessed during query-time processing. It would be helpful if the authors could clarify the storage mechanism and how the associated data-loading overhead is accounted for in the latency evaluation.
4 comments

Response to Reviewer F4Wb

Thank you for your careful reading and constructive suggestions. Your recognition of our clear motivation, higher supported workload, and practical, training-free design is a great encouragement to us!

Following your suggestions, we add clustering comparisons and qualitative examples, and clarify our contribution and frame-access costs.

W1 (Part 1). Core contribution

StreamIndex's core contribution is identifying and addressing two scalability bottlenecks in existing streaming video methods, as illustrated in Figure 1 (right): continuous visual encoding can exceed the available compute budget, while accumulating encoded history can exhaust GPU memory. Both arise from encoding frames before knowing their relevance to a query.

StreamIndex uses one representative index that supports both retrieval and reuse: representative embeddings identify query-relevant clusters, and cached representative features support selective feature reuse during query-time encoding. This unified design decouples stream-time indexing from query-time visual encoding to balance answer accuracy and computation cost, thus supporting multi-stream scalability.

W1 (Part 2). Clustering ablation

To evaluate cluster segmentation, we compare histogram clustering with fixed-length strategies (Fixed-4, Fixed-6, and Fixed-8), which group every 4, 6, and 8 consecutive sampled frames into one cluster, respectively. The comparison on OVO-Bench is included in Table 6 of the revised manuscript.

SegmentationReal ↑Bwd. ↑Fwd. ↑Total ↑Total Enc. Frames ↓
Histogram (StreamIndex)71.8446.4139.5352.5938.33
Fixed-472.1545.9340.4652.8457.80
Fixed-672.1244.8039.7552.2240.30
Fixed-871.8245.5038.9752.0031.88

Finer fixed-length segmentation improves accuracy but increases encoding cost, consistent with the expected trade-off: from Fixed-8 to Fixed-4, Total accuracy rises from 52.00% to 52.84%, while Total Encoded Frames increases from 31.88 to 57.80. StreamIndex's dynamic histogram-based segmentation balances accuracy and total computation cost.

Q1. Success and failure cases

We have added qualitative examples in Figures 8–9. Figure 8(a,d) shows successful evidence preservation and correct color recognition; (c) answers correctly with partial evidence. In contrast, (b) misses the clusters containing lamp-state evidence, while (e) retrieves the relevant cluster but selects frames after the target disappears. Thus, subtle state evidence can be missed at retrieval, and successful cluster retrieval does not ensure evidence retention.

Figure 9 compares three frame-selection rules: adjacent-frame differences (StreamIndex default), first-to-current differences, and cumulative adjacent-frame differences. No rule consistently preserves the most evidence across all three cases: adjacent-frame differences retain more evidence in (C), while cumulative differences retain more in (B) but miss it in (A) and (C).

Since relevant evidence can occur anywhere within a cluster, no fixed, query-agnostic frame-selection heuristic can guarantee evidence retention across all scenarios. StreamIndex offers a practical heuristic with a failure boundary.

Q2. Frame storage and access cost

StreamIndex stores decoded frame pixels in CPU memory and cluster representative embeddings in GPU memory.

  • During stream ingestion, recent sampled frames stay in a FIFO buffer, while older frames are organized into clusters and retained in CPU memory. Each cluster's representative is visually encoded, and its embedding remains on the GPU as part of the retrieval index.
  • When a question arrives, query–representative similarity is computed on the GPU to retrieve relevant clusters. The system then accesses their frame pixels in CPU memory, selects frames, and transfers the required visual inputs to the GPU for on-demand encoding. This avoids reloading the representative embeddings or reopening and decoding historical video for each query.

Frame access and CPU-to-GPU transfer are included in our end-to-end query latency, which covers retrieval, query-time encoding, and generation. In an OVO-Bench measurement, historical-frame access averaged 0.0097 ms, accounting for 0.0011% of query latency. Transferring historical cluster frames from CPU to GPU accounts for only 0.28% of the full end-to-end latency, indicating a small transfer overhead.

Summary

In summary, thank you again for helping us clarify our contribution, strengthen the clustering evaluation, illustrate successes and failures, and explain storage and latency accounting.

We hope our responses have addressed your concerns and would sincerely appreciate further feedback and your reconsideration of the rating in light of the additional evidence.

Dear Reviewer F4Wb,

Following up on our rebuttal, we wanted to kindly check whether you might have any further concerns or feedback to share.

Your suggestions have helped us improve the presentation and evaluation of our work. In our response, we have further clarified the contribution of our framework, provided additional clustering comparisons and qualitative success and failure examples, and explained historical-frame storage and how frame-access and transfer costs are included in our latency measurements.

We greatly appreciate your time and consideration. Please feel free to share any thoughts you may have; we would be happy to provide further clarification if helpful.

Best regards,

The Authors

Thank you for the author's explanation, which has resolved some of my doubts. I will raise my score, but I may not be confident.

Dear Reviewer F4Wb,

Thank you very much for your positive feedback and willingness to raise your rating. We sincerely appreciate your recognition of our work and your constructive suggestions. In the final version, we will carefully incorporate your comments and make further revisions to clarify the novelty and technical contributions of StreamIndex, strengthen the completeness of our experimental analysis.

Your feedback has been invaluable in helping us improve the quality and clarity of our work. Thank you again for your encouragement and support!

Best regards, The Authors

Reviewer_DjtZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

StreamIndex is a training-free framework for serving concurrent video streams under a fixed GPU budget. It reduces continuous visual encoding by clustering incoming frames using CPU-side grayscale histogram statistics and encoding one representative per cluster. At query time, the system retrieves relevant clusters, selects low-motion frames, and recomputes features at selected spatial positions while reusing features from the cluster representative. Summaries of unretrieved intervals and recent frames provide additional evidence for response generation. The evaluation combines accuracy on OVO-Bench and StreamingBench with separate experiments on stream concurrency, resolution, history length, and query load. The main contribution is the reallocation of computation between continuous ingestion and query answering, rather than a uniform improvement in both accuracy and response latency across all settings.

Critical strengths
  1. A practically relevant problem and a coherent overall design. The paper identifies continuous visual encoding as an important cost when multiple video streams share a GPU. Separating inexpensive ingestion from selective query-time computation is a sensible way to exploit workloads in which queries are relatively infrequent. The components are consistent with this objective, and the framework requires no additional training, making it practically useful.

  2. Substantial reported serving-capacity gains under controlled workloads. The experiments examine concurrency, resolution, history length, and query arrival rate, rather than focusing exclusively on single-stream inference speed. In Table 2, at eight streams and four queries per minute per stream, StreamIndex reports P50/P95 latency of 1.90/4.34 seconds, compared with 4.51/7.17 for HERMES and 20.53/31.14 for ReKV. At 16 streams and two queries per minute per stream, its P95 is 5.57 seconds, while HERMES violates the real-time constraint and ReKV runs out of memory. The heterogeneous-stream evaluation provides useful evidence beyond the homogeneous stress tests.

  3. The paper clearly reports performance trade-offs and implementation limitations. The evaluation covers both Qwen and LLaVA-OneVision backbones. The authors acknowledge that pre-encoded memory can offer lower latency under light workloads, that HERMES achieves higher accuracy on LLaVA-OV-7B, and that StreamIndex also fails to meet the real-time constraint at the highest tested query load. Appendix B.3 explicitly describes the changes to positional and attention computation introduced by sparse encoding. These qualifications help readers understand the findings and their scope.

Critical weaknesses
  1. The motion statistic does not directly measure whether the cached reference remains valid. Equations (2)-(4) construct the reuse mask from differences between adjacent frames, averaging the motion slices associated with the selected frames. Equation (5), however, reuses features from the cluster's first frame. Small adjacent-frame changes do not imply similarity to that first frame. For example, an object can move within a cluster and then become stationary: the selected stationary frames can have very low local motion even though the object's state or location differs from the cached representative. A grayscale histogram can also remain similar throughout such a change. Averaging only the motion slices associated with selected frames can omit the earlier transition, and the preference for low-motion frames can reinforce this failure mode.

  2. The experiments do not isolate the benefits of the core indexing and retrieval design from those of simpler sparse-ingestion alternatives. Table 4 ablates gap summaries, frame selection, and token reuse, but retains the underlying clustering and retrieval pipeline. The paper does not compare histogram-based clustering with fixed-duration segmentation under comparable representative-encoding and query-time budgets. It also does not establish how much historical retrieval contributes beyond recent context. Consequently, the results demonstrate the usefulness of reducing continuous encoding, but do not sufficiently establish why this particular index is preferable to a simpler periodic-keyframe index with the same on-demand recovery budget.

  3. Relevant prior work limits the component-level novelty and calls for more precise positioning. The distinction in Section 2 between feature-guided compression and selection before visual encoding is too broad. GRT, described in DIVE-Bench: Codec-Style High-FPS Video Understanding, already uses pixel-level changes to gate static patches before expensive visual processing without retraining [1]. Deja Vu/ReuseViT also investigates inter-frame computation reuse for video-language workloads, although it uses learned reuse mechanisms [2]. These works do not establish that StreamIndex's complete streaming framework has already been proposed, but they are directly relevant to positioning its selective-encoding component.

Questions for authors
  1. Reference consistency in feature reuse: Equations (2)-(4) use adjacent-frame differences to determine reuse positions, whereas Equation (5) reuses features from the cluster's first frame. How does this criterion avoid treating regions with small adjacent-frame changes but substantial deviation from the first frame as reusable? In particular, can you analyze motion-then-stabilization and gradually changing states, and compare against representative-to-current differences or cumulative-change masks under the same reuse budget?

  2. Independent benefits of the core indexing design: With the same number of representative-frame encodings and the same query-time detail-recovery budget, how does histogram-based clustering compare with fixed-duration segmentation or periodic-keyframe indexing in accuracy and end-to-end latency? How much does historical retrieval improve over a recent-frame-only configuration? Please provide controlled comparisons that distinguish the contributions of the specific indexing and retrieval design from the benefits of reducing continuous encoding itself.

  3. Distinctions from related work: GRT already filters static patches using pixel-level changes before expensive visual computation, and Deja Vu/ReuseViT also studies inter-frame computation reuse for video-language tasks. What substantive technical differences distinguish the proposed selective encoding and feature reuse from these methods? Please revise the characterization of prior work in Section 2 accordingly and clarify which innovations concern the individual components and which arise from their integration with streaming indexing, query-time recovery, and multi-stream serving.

4 comments

Response to Reviewer DjtZ

Thank you for your careful reading and constructive suggestions. We appreciate your recognition of the practical relevance and serving-capacity gains of StreamIndex, which is a great encouragement to us!

Following your suggestions, we provide additional clustering and historical-retrieval comparisons, discuss reference consistency, and clarify our distinctions from prior work below.

W1 & Q1. Reference consistency

Our adjacent-frame reuse criterion does not guarantee consistency with the cached first-frame features. Motion followed by stabilization or gradual changes can leave low adjacent-frame differences despite substantial deviation from the reference. Histogram clustering mitigates this issue by restricting reuse to frames within the same cluster.

For spatial reuse, Figure 10 (Appendix F.3) compares adjacent-frame, first-to-current, and cumulative differences with selected frames and reuse budgets fixed. First-to-current masks yield lower pixel discrepancy in the illustrated fused image proxies.

For temporal frame selection, Figure 9 illustrates the risk of missing evidence with each fixed strategy. No training-free method can reserve evidence in every case. StreamIndex adopts a motion-then-stable heuristic as a practical trade-off between evidence preservation and computational efficiency.

W2 & Q2 (Part 1). Clustering Strategy Ablation

To evaluate cluster segmentation, we compare histogram clustering with fixed-length strategies (Fixed-4, Fixed-6, and Fixed-8), which group every 4, 6, and 8 consecutive sampled frames into one cluster, respectively. The comparison on OVO-Bench is included in Table 6 of the revised manuscript.

SegmentationReal ↑Bwd. ↑Fwd. ↑Total ↑Total Enc. Frames ↓Query-Time Enc. Frames ↓Query-to-Resp. Latency (s) ↓
Histogram (StreamIndex)71.8446.4139.5352.5938.335.200.71
Fixed-472.1545.9340.4652.8457.805.510.83
Fixed-672.1244.8039.7552.2240.305.500.78
Fixed-871.8245.5038.9752.0031.885.490.77

Compared with Fixed-6, the configuration with the closest encoding budget, histogram clustering achieves higher accuracy on both Backward tasks and overall. These results demonstrate that StreamIndex's dynamic histogram-based segmentation balances accuracy and total computation cost.

W2 & Q2 (Part 2). Historical retrieval

SimpleStream [1] reports competitive accuracy on OVO-Bench using only four recent frames. To more directly assess the value of historical retrieval, we additionally evaluate on OVO-S-Bench [2], a streaming spatial intelligence benchmark whose context-tracking tasks require evidence no longer visible at query time. We compare recent four-frame context with and without top-4 cluster retrieval (at most 16 frames per cluster) against uniform 128-frame sampling.

MethodContext Learning Accuracy (%) ↑
Recent 4 frames only35.00
Recent 4 frames + cluster-based historical retrieval45.19

With the same four-frame recent context, adding cluster-based historical retrieval raises accuracy from 35.00% to 45.19% (+10.19 percentage points). This gain shows that StreamIndex’s accuracy benefits from historical retrieval beyond recent context alone.

W3 & Q3. Related-work distinctions

We have revised Section 2 (Related Work) to acknowledge GRT and Déjà Vu/ReuseViT and clarify that pre-encoding filtering and inter-frame reuse alone are not our novelty.

Unlike GRT’s pixel-level gating and scene-token merging or ReuseViT’s learned reuse, StreamIndex integrates training-free selective encoding with query-agnostic streaming indexing. One representative index supports both retrieval and reuse, deferring historical detail encoding to retrieved clusters to reduce continuous GPU computation and control query-time cost.

Summary

In summary, we thank you again for your valuable suggestions and consideration. They have helped us strengthen the evidence for clustering and historical retrieval, clarify the limitations of motion-based reuse, and better articulate the novelty of our integrated streaming design.

We hope our detailed responses have addressed your concerns and would sincerely appreciate any further feedback.

References

[1] A Simple Baseline for Streaming Video Understanding. arXiv:2604.02317, 2026.

[2] OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs. EMNLP 2026.

Dear Reviewer DjtZ,

Following up on our rebuttal, we wanted to kindly check whether you might have any further concerns or feedback to share.

Your suggestions have helped us strengthen our analysis and better explain our design choices. In our response, we have provided additional clustering and historical-retrieval comparisons, discussed the reference-consistency limitation of feature reuse, and clarified the contribution of our integrated streaming design in relation to GRT and Déjà Vu/ReuseViT.

We greatly appreciate your time and consideration. Please feel free to share any thoughts you may have; we would be happy to provide further clarification if helpful.

Best regards,

The Authors

Thank you for the detailed and constructive responses. The authors have adequately addressed my major concerns through additional experiments and clarifications. In particular, the comparisons with fixed-length segmentation provide stronger evidence for the proposed histogram-based indexing strategy, while the historical retrieval evaluation demonstrates its contribution beyond recent-frame context. The discussion of reference consistency and the clarification of differences from GRT and ReuseViT also improve the technical positioning of the work. Overall, I am satisfied with the rebuttal, and the revised manuscript provides sufficient evidence to support the main claims. I therefore consider my previous concerns resolved.

Dear Reviewer DjtZ,

Thank you for your thoughtful feedback and for taking the time to review our additional experiments and clarifications. We’re glad they resolved your concerns. Thank you again for your encouragement and support!

Best regards, The Authors

Reviewer_crTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper motivates its coarse indexing and EMA gap summarization as a way to avoid continuous visual computation. This prevents unnecessary GPU visual encoding before query arrival. During video ingestion, incoming frames are clustered using CPU-side grayscale histogram statistics, caching only one encoded representative frame per cluster. When a user query arrives, the system retrieves relevant clusters and constructs a spatio-temporal motion tensor on the CPU. This motion tensor guides low-motion frame selection and spatial token reuse. Only the selected frames are encoded with spatial tokens being reused from cached results. The framework operates without additional model tuning and demonstrates compatibility across multiple open-source MLLM families.

Critical strengths

The system performs motion estimation to the CPU using purely pixel-level frame differences and smoothing operations and uses the CPU-computed motion tensor to drive both temporal frame selection and spatial token reuse.

The filtering is simple and is done before the ViT encoding to save computation and memory. The idea to decouple indexing from encoding is bold.

Deriving both temporal frame selection and spatial token reuse from a single, consistent spatio-temporal motion tensor is interesting.

Critical weaknesses

The CPU filtering and low-motion frame selection is fast. But it has its own weakness in that it removes frames before the model can before the model can understand fine-grained information or dynamic actions in the video. When does it perform better (e.g., static scenes with high visual redundancy) vs. when is this not a good idea (e.g., fast-paced sports, camera transitions, or action-critical events)?

Recent streaming memory literature, InfiniPot-V (NeurIPS 2025), StreamingToM, and ProtoKV (ICML 2026), has actively explored training-free, query-agnostic token retention and summarization techniques. These are very closely related and more recent than ReKV, but they are not explicitly compared. This makes it unclear as to whether StreamIndex's on-demand recovery truly outperforms existing approaches under realistic queries.

The framework indexes only the first frame of each cluster as its representative embedding and compresses unselected temporal gaps using a simple chronological EMA across representative tokens. This design lacks sufficient representational capacity to preserve diverse intra-cluster semantics or fine-grained historical events. As evidenced by Table 3 and Table 6, the performance on Backward and Forward temporal reasoning shows notable degradations compared to stronger memory baselines like HERMES.

The system is baised toward low-motion frames and this may make the system inherently vulnerable to action-oriented queries. (See below)

Questions for authors
  • How does it compare with recent work, such as InfiniPot-V, StreamingToM, and ProtoKV? All of them are training-free. Specifically, under extended video lengths and fixed GPU memory budgets, what is the accuracy trade-off when answering queries that refer to fine-grained past events?

  • Prior works (e.g., ProtoKV) argue that preserving only coarse representatives leads to severe evidence loss for delayed queries. Given that StreamIndex only encodes the initial frame per cluster and uses a single EMA vector for unselected gaps (Eq. 8), how robust is the retrieval when the answer-bearing evidence lies in the tail of a cluster or within skipped intervals? Can you provide an evidence retrieval recall evaluation under varying query delays?

  • Accuracy of clustering. The cluster segmentation is driven purely by low-level grayscale histograms without semantic awareness. Is there direct empirical evidence that the top-kk retrieved clusters actually contain the ground-truth evidence frames needed for the query? What is the Evidence Retrieval Recall compared to embedding-based retrieval or uniform downsampling across the stream history?

  • In Table 1 and Table 2, baselines like HERMES and ReKV are tested under a single FIFO worker sharing GPU compute for both ingestion and queries. If these baselines adopt simple heuristic stream-time frame skipping (e.g., 0.2 FPS or keyframe-only ingestion), how does StreamIndex's latency and accuracy compare?

  • StreamIndex uses a grayscale histogram distance for clustering and Gaussian-smoothed frame difference for the motion tensor. In edited videos (e.g., YouTube vlogs, gaming streams) with rapid camera cuts, lighting shifts, digital noise, or continuous camera motion, won't the histogram distance cause severe over-segmentation (creating too many single-frame clusters)?

OVO-Bench and StreamingBench don't have much high motion segments. In real-world streaming (e.g., sports broadcasting, autonomous driving dashcams, action cameras, and professionally edited YouTube content), the critical events queried by users typically occur within highly dynamic intervals characterized by rapid actions or transitions. Your system bias frame selection toward low-motion frames which makes the system inherently vulnerable to action-oriented queries. This limitation is empirically reflected in Table 3 and Table 4, where low-motion selection yields stagnant or degraded accuracy on forward reasoning tasks. This raises a fundamental concern that the reported performance gains may largely stem from the static bias of the evaluated benchmarks.

  • The CPU filtering and low-motion frame selection is fast. But it has its own weakness in that it removes frames before the model can understand whether those dropped frames contain the exact evidence needed for the query. The paper doesn't explore the tradeoff. When does it perform better vs. when is this not a good idea?

  • How does it handle bursty query? The computation is delayed until the query time. So handling bursty query might be more challenging.

6 comments

Thank you for your careful reading and detailed, constructive suggestions. Your recognition of the idea of decoupling indexing from encoding and your interest in using a shared motion tensor for frame selection and token reuse are a great encouragement to us!

Following your suggestions, we provide additional baseline comparisons and analyses of delayed queries, evidence preservation, and serving behavior, while clarifying the limitations of our approach.

W2 & Q1. Recent memory baselines

StreamingTOM [1] retrieves query-relevant groups from quantized historical KV memory, placing it in Retrieval-from-Memory alongside ReKV. InfiniPot-V [2] continually compresses a query-agnostic KV cache and directly uses the retained cache to answer queries, placing it in Memory-as-Context alongside HERMES. Although StreamingTOM reduces tokens before the LLM, its selection operates on already-encoded visual features; both methods therefore retain the encode-first design discussed in our paper.

We compare these methods on OVO-Bench with Qwen2.5-VL-7B at 1 FPS. StreamingTOM and InfiniPot-V use local runs with their default configurations. The three task scores below are taken from Table 7.

MethodReal-time ↑Backward ↑Forward ↑Total ↑
ReKV63.6546.1639.2349.68
StreamingTOM61.5044.3952.6652.85
HERMES63.0646.6742.9050.88
InfiniPot-V64.7546.3646.7252.61
StreamIndex71.8446.4139.5352.59

Extended-history evaluation under a fixed GPU memory budget remains unaddressed by these accuracy comparisons. ProtoKV [3] was not open-sourced at the time of our evaluation, so we did not reproduce it locally or make an empirical performance claim against it. As discussed in Section 1, StreamIndex trades continuous stream-time encoding for on-demand detail recovery at query time, yet retains competitive accuracy relative to both memory paradigms.

W3 & Q2. Delayed-query evidence

To examine how query delay affects access to evidence, we use OVO-S-Bench [4], whose L1 (Instantaneous Egocentric Perception) and L2 (Spatiotemporal Context Tracking) tasks distinguish current-view perception from historical context tracking. This distinction helps assess how delayed queries affect these two types of evidence.

In StreamIndex, evidence can be lost through (1) frame selection or token reuse within a retrieved cluster, or (2) exclusion of the evidence-bearing cluster from the top-k, leaving only a coarse gap summary. For completed historical clusters, delay leaves the within-cluster rules unchanged but adds competing clusters.

  • L1 evidence can leave the recent-frame buffer and become dependent on historical retrieval and selective recovery.
  • L2 already relies on historical evidence, so delay mainly affects cluster selection.
Query delay (s)L1 ↑L2 ↑L1 change vs. 0 s (pp)L2 change vs. 0 s (pp)
040.0645.19——
6031.9040.92−8.16−4.27
12033.2639.46−6.80−5.73

At delays of 60 and 120 seconds, L1 drops by 6.80–8.16 percentage points, compared with 4.27–5.73 for L2. L1 declines more at both delays. These results do not directly isolate retrieval misses. The larger L1 decline is consistent with losing recent-frame access, while the L2 decline is consistent with increased competition for historical cluster selection.

References

[1] StreamingTOM: Streaming Token Compression for Efficient Video Understanding. CVPR 2026.

[2] InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding. NeurIPS 2025.

[3] ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory. ICML 2026.

[4] OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs. EMNLP 2026.

Q3 (Part 1). Clustering Segment Strategy

To evaluate cluster segmentation, we compare histogram clustering with fixed-length strategies (Fixed-4, Fixed-6, and Fixed-8), which group every 4, 6, and 8 consecutive sampled frames into one cluster, respectively. The comparison on OVO-Bench is included in Table 6 of the revised manuscript.

SegmentationReal ↑Bwd. ↑Fwd. ↑Total ↑Total Enc. Frames ↓
Histogram (StreamIndex)71.8446.4139.5352.5938.33
Fixed-472.1545.9340.4652.8457.80
Fixed-672.1244.8039.7552.2240.30
Fixed-871.8245.5038.9752.0031.88

Finer fixed-length segmentation improves accuracy but increases encoding cost, consistent with the expected trade-off: from Fixed-8 to Fixed-4, Total accuracy rises from 52.00% to 52.84%, while Total Encoded Frames increases from 31.88 to 57.80. StreamIndex's dynamic histogram-based segmentation balances accuracy and total computation cost.

W3 & Q3 (Part 2). Historical QA and retrieval recall

StreamIndex maintains competitive historical QA accuracy, although this does not directly measure Evidence Retrieval Recall.

We compare StreamIndex with uniform temporal sampling and ReKV's historical KV-cache retrieval on OVO-Bench using Qwen2.5-VL-7B. The results below are extracted from Table 7: uniform sampling uses a video-length-dependent rate, while ReKV and StreamIndex use 1 FPS.

MethodFrame rateBackward Accuracy (%) ↑
Uniform temporal samplingAdaptive (1.0 / 0.5 / 0.2 FPS)42.48
ReKV1 FPS46.16
StreamIndex1 FPS46.41

StreamIndex improves Backward accuracy over uniform sampling by 3.93 percentage points and performs similarly to ReKV. These results support competitive end-to-end historical QA, but a direct Evidence Retrieval Recall comparison with uniform sampling or embedding-based retrieval remains unreported.

Q4. Sparse-sampling trade-offs

Sparse frame sampling reduces baseline latency, while StreamIndex at 1 FPS achieves the lowest P50/P95 latency in both tested workloads. We compare HERMES and ReKV at 0.2 and 1 FPS with StreamIndex at 1 FPS. Entries report P50/P95 end-to-end query-to-response latency in seconds, including queueing; query rates are per stream. RT fail denotes a real-time-bound violation, and OOM denotes an out-of-memory failure.

Method8 streams × 4 queries/min16 streams × 2 queries/min
HERMES (0.2 FPS)3.44 / 5.198.49 / 13.82
HERMES (1 FPS)4.51 / 7.17RT fail
ReKV (0.2 FPS)2.39 / 4.59OOM
ReKV (1 FPS)20.53 / 31.14OOM
StreamIndex (1 FPS)1.90 / 4.342.78 / 5.57

At 8 streams, reducing ReKV's sampling rate to 0.2 FPS lowers its P95 latency from 31.14 to 4.59 s, still above StreamIndex's 4.34 s at 1 FPS. At 16 streams, StreamIndex achieves a P95 latency of 5.57 s versus 13.82 s for HERMES at 0.2 FPS, while ReKV runs out of memory at both sampling rates. Sparse sampling narrows the latency gap, but StreamIndex retains lower latency at a higher sampling rate and avoids the failures observed for the baselines in these workloads.

Q5. Over-segmentation

We agree that frequent changes in the global grayscale distribution can cause over-segmentation and reduce StreamIndex's efficiency gains. Our clustering compares each frame's normalized grayscale histogram with that of the active cluster's first frame. An isolated camera cut or lighting shift may start a new cluster, but subsequent frames can join it if their histogram distance remains below the threshold and the cluster size limit is not reached.

In our OVO-S-Bench[1] L1/L2 evaluation, singleton clusters account for 23% of all clusters, and 0% of videos consist entirely of singleton clusters. This confirms that singleton clusters occur in practice, while most clusters still contain multiple frames.

W1, W4 & Q6. Low-motion selection trade-offs

We select OVO-S-Bench as an additional benchmark specifically to address the reviewer's concern about the static bias of the evaluated benchmarks.

  • L1, Instantaneous Egocentric Perception, includes camera/object motion perception
  • L2, Spatiotemporal Context Tracking, tests memory of spatial evidence no longer visible at query time.

These tasks provide complementary evidence beyond the original evaluations. Baseline results are reported by OVO-S-Bench; StreamIndex is evaluated locally.

MethodL1 ↑L2 ↑
Uniform sampling (128 frames)40.6745.51
HERMES40.8845.41
InfiniPot-V39.0535.72
StreamingToM37.2448.18
StreamIndex40.0645.19

Result show that StreamIndex achieves balanced and competitive accuracy across L1 and L2.

Low motion means small changes relative to the preceding sampled frame, not a static event. Dynamic sequences can contain stable, informative views: Figure 9(C) retains two evidence frames, with a correct answer in Figure 8(c). StreamIndex adopts a training-free motion-then-stable heuristic as a practical trade-off between evidence preservation and computational efficiency, without guaranteeing that all relevant evidence is retained in every case.

Q7. Handling bursty queries

StreamIndex handles the query bursts with sublinear growth in burst-processing time. Our implementation supports concurrent serving and SGLang generation batching. We submit 1–16 simultaneous requests and report burst-processing time and its ratio to the single-request case.

Simultaneous requests124816
StreamIndex time1.902.043.096.9611.67
Ratio to single request1.00×1.08×1.63×3.67×6.15×

Across 2–16 requests, measured burst-processing time remains below the theoretical serial reference. At 16 requests, it is 6.15× the single-request time, versus 16× for serial processing. These results support burst-handling efficiency.

Summary

In summary, thank you again for your valuable suggestions and consideration. They have helped us strengthen the comparisons and clarify StreamIndex’s accuracy–scalability trade-offs, including limitations under delayed queries and action-critical evidence, as well as its efficiency under the tested query bursts. We have not conducted a direct Evidence Retrieval Recall evaluation; our empirical conclusions concern end-to-end answer accuracy.

We hope our responses have addressed your concerns and would sincerely appreciate further feedback and your reconsideration of the rating in light of the additional evidence.

References

[1] OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs. EMNLP 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and reviewers crTn (R1), DjtZ (R2), and F4Wb (R3) for their feedback. We appreciate R1’s positive assessment of StreamIndex’s design and the practical value that R2 and R3 highlighted for multi-stream deployment.

We answered R1’s follow-up questions about baseline settings, accuracy gain beyond recent frames, and how StreamIndex performs relative to the baselines. We are grateful that R2 and R3 found our responses addressed their concerns. R2 maintained a rating of 3, and R3 raised their rating to 3. We summarize our contributions and rebuttal findings below. The revised PDF highlights changes in red.

Core contributions

  1. Bottlenecks in multi-stream deployment: Both baseline families encode frames before selecting evidence. Memory-as-Context methods maintain a compact encoded history for direct input to the LLM. With multiple streams on one GPU, HERMES becomes compute-bound because continuous encoding and memory updates cannot keep pace with live input. Retrieval-from-Memory methods store encoded history for retrieval when a query arrives. ReKV becomes memory-bound as the accumulated history exhausts GPU memory (Figure 1). Across resolutions and stream counts, Figure 4 shows out-of-memory failures on Qwen2.5-VL-7B and growing processing lag on LLaVA-OV-7B.

  2. Stream-time indexing and query-time encoding: StreamIndex builds one representative index that supports both retrieval and reuse. During ingestion, it groups similar frames on the CPU and encodes one representative per finalized cluster. At query time, representative embeddings identify relevant clusters, and cached representative features reduce the cost of encoding selected frames.

  3. Competitive accuracy and scalable query serving: StreamIndex maintains competitive accuracy on OVO-Bench and StreamingBench. With 16 concurrent 1080p streams and two queries per minute per stream, it sustains real-time ingestion with 5.57 s P95 response latency. At the same load, HERMES misses the real-time bound and ReKV exhausts GPU memory (Table 2). For burst queries, our rebuttal experiment shows sublinear growth in total processing time as burst queries size increases (R1-Q7).

Updates on Clarifications and Experiments During Rebuttal

  1. Comparison with additional baselines (StreamingTOM and InfiniPot-V) (R1-W2/Q1, R1-Q4, R1 follow-up on recent baselines). At R1’s request, we evaluated StreamingTOM and InfiniPot-V, both of which keep low memory costs and comparable accuracy. However, with multi-stream settings (e.g. 16 streams, 2 queries/min per stream), both violate the real-time bound. Under the same load, HERMES also violates the bound at 1 FPS and ReKV exhausts GPU memory (Table 2). Across all methods tested, only StreamIndex sustains end-to-end query latency of 2.78 s at P50 and 5.57 s at P95.

  2. Adaptive clustering vs. fixed-length clustering (R1-Q3, R2-W2/Q2, R3-W1). At a comparable encoding budget, adaptive clustering achieves higher OVO-Bench accuracy and lower query-to-response latency than fixed-length clustering (Appendix B.1, Table 6).

  3. StreamIndex cluster improves accuracy beyond simple recent frames (R1-W3/Q2, R1 follow-up, R2-W2/Q2): On OVO-S-Bench L2, adding historical clusters to recent frames raises accuracy from 35.00% to 45.19% (+10.19 pp). Delays of 60 and 120 seconds reduce accuracy by 4.27 and 5.73 pp, respectively. Both delayed results exceed the recent-frames-only baseline, supporting StreamIndex’s cluster design.

  4. Baseline settings (R1 follow-up): LLaVA-OV-7B pools each frame into a fixed 196 visual tokens. Qwen2.5-VL-7B uses dynamic resizing, so its token count depends on the input resolution. To keep the per-frame budgets comparable for benchmarking, we set Qwen’s max_pixels to 360 × 420 for both StreamIndex and the baselines, yielding 192 visual tokens per 4:3 frame (Appendix A).

  5. Qualitative evidence on frame selection and feature reuse (R1-W1, R1-W4/Q6, R2-W1/Q1, R3-Q1): Retrieval misses the relevant cluster in Figure 8(b). Frame selection misses the target after successful retrieval in Figure 8(e). Evidence timing affects frame selection (Figure 9), and comparison with the cached representative reduces pixel discrepancy in the illustrated reuse cases (Figure 10).

  6. Historical-frame access (R3-Q2): Decoded frames reside on the CPU and retrieval embeddings on the GPU. Historical-frame transfers account for only 0.28% of end-to-end query latency on OVO-Bench.


We believe these experiments and clarifications address the reviewers' concerns and enhance the quality of our manuscript. We are grateful for the reviewers' valuable suggestions and the Area Chair's diligent oversight.

With warm thanks,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.