acceptodds
Under review as a conference paper at ICLR 2027

ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

Abstract

Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods largely focus on accelerating individual modules, such as visual encoding, token pruning, or KV-cache compression, but provide limited insight into whether the resulting system can sustain real-time streaming performance. We formulate streaming VideoLLM inference as a coordinated pipeline spanning visual preprocessing, visual encoding, token dropping, and LLM prefilling/decoding. Building on this formulation, we propose ViCoStream (Video Coordinated Streaming), a stage-wise coordinated streaming framework that combines chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs. We further provide a systematic study of bottleneck migration, revealing how chunk size, token retention, attention locality, and retrieval scope shape the throughput–accuracy trade-off. Experiments with Qwen2.5-VL-3B/7B-Instruct across multiple streaming benchmarks show that ViCoStream achieves 134 FPS video throughput and <50ms TTFT on a single A100 GPU while maintaining accuracy close to full-history baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Language Models, Real-Time Video Understanding, Streaming Inference, Visual Token Compression, Coordinated Parallel Streaming, Efficient Video Processing, Attention Locality, Query-Driven Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aYj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper treats continuous VideoLLM inference as a pipeline over visual preprocessing, visual encoding, token dropping, and LLM prefilling and decoding. It combines chunked execution and CUDA-stream overlap with temporal token dropping, local visual attention, and query-side retrieval, and fine-tunes Qwen2.5-VL-3B/7B with supervised instruction tuning under an adaptive visual-scope mask. Component profiling and evaluations on six streaming-oriented benchmarks report a calculated maximum of 134 FPS, query TTFT below 50 ms, and accuracy near full-history streaming under moderate compression.

Critical strengths
  • Chunking amortizes visual-encoder calls, local attention bounds the LLM work for each incoming chunk, and retrieval restores selected older evidence. These parts address different bottlenecks in one pipeline, while Equation (3) adapts training to the same restricted attention scopes used at inference.

  • One 3B configuration in Table 3 drops 57.6% of the visual tokens while improving StreamingBench by 0.12 points, StreamBench by 1.35 points, and RVS-Ego by 0.63 points. Its losses on OVO-Bench and RVS-Movie remain below two points, showing that moderate compression can provide a useful trade-off on this backbone.

Critical weaknesses
  • The technical novelty is limited. Section 4 combines chunk-wise encoding, CUDA-stream preprocessing, similarity-based token dropping, local attention, query retrieval, and a matching training mask, but none of these components introduces a substantial new algorithm. The paper therefore needs strong end-to-end evidence that coordinating them produces a system-level benefit beyond a straightforward combination of existing techniques.

  • It is not fully clear whether the reported 134 FPS represents sustained wall-clock throughput of the complete pipeline or an estimate based on stage-wise latency. In the c=16c=16, 30%-retention row of Table 2, the number is consistent with 16,000/11916{,}000/119 ms, where 119 ms is the slowest measured stage. Clarifying the measurement procedure and reporting a sustained concurrent run or pipeline trace would make this efficiency result easier to assess, especially because the GPU stages share one A100.

  • The current component studies make it difficult to isolate the benefit of coordination. Table 5(b) focuses on preprocessing, while Table 5(c) changes cc, acn, the dropping rule, and the first-frame anchor without matching the actual retained token count or prefill latency. A serial-versus-pipelined end-to-end control and an ablation of the adaptive training mask would help clarify how much each part contributes.

  • It is difficult to connect the headline efficiency and accuracy results at exactly the same operating point. Table 2 reports 134 FPS for c=16c=16 and 30% token retention, while Tables 3 and 4 vary acn, ur, and sdrops_{\mathrm{drop}} without reporting the corresponding cc, FPS, or TTFT. The s=0.10s=0.10 rows closest to 30% retention show noticeable drops on several 7B benchmarks, although they may not be the exact configuration used for the 134-FPS result. Reporting accuracy and efficiency together for that configuration would make the central trade-off clearer.

Questions for authors
  1. A sustained wall-clock run of the complete pipeline under the c=16c=16, 30%-retention setting would be helpful. Reporting total processed frames, elapsed time, and a serial or pipeline-disabled control would show whether the derived 134 FPS is achieved in practice.

  2. Adding chunk size, streaming FPS, and query TTFT to Tables 3 and 4 would allow accuracy and efficiency to be compared at the same operating point. Table 4 would also benefit from the measured token retention already reported for Table 3, especially for the exact configuration used in the 134-FPS result.

  3. Table 5(c) would be more informative if the frame-level and chunk-level rows matched the retained token count, visible temporal span, and LLM prefill latency. This would separate the effect of chunk-level dropping from the additional context or tokens retained by each configuration.

5 comments

We thank the reviewer for recognizing the complementary roles of chunking, local attention, retrieval, and training-time visibility. We agree that the systems contribution requires direct execution controls and careful ablations.

W1: The technical novelty is limited. ... The paper therefore needs strong end-to-end evidence that coordinating them produces a system-level benefit beyond a straightforward combination of existing techniques.

We thank the reviewer for stating this concern directly. We position the contribution as the stage-wise formulation and analysis of continuous VideoLLM inference, including how reducing LLM-side work exposes front-end bottlenecks. The individual uses of CUDA streams, token dropping, and local attention are not claimed as algorithmic novelty and coordinating them changes throughput. To test that claim in isolation, we run the same modules under three execution modes (serial, preprocessing-overlapped, and four-stage) in the following study.

The table reports wall-clock throughput on one A100-SXM4-80GB and is averaged over two runs per setting.

ConfigurationFrames/runSerial FPSVP-overlap FPSFour-stage FPSSpeedup
Base 3B, 448²-pixel budget, 16 frames, fixed 30%256052.3869.0993.351.78×
Fine-tuned 3B, 90,000-pixel budget, 16 frames, fixed 30%256077.80114.13159.302.05×
Fine-tuned 3B, 90,000-pixel budget, 8 frames, s=0.25128058.4480.6792.791.59×

Timing covers startup/drain, preprocessing, transfers, encoding, dropping, and LLM execution including the output projection, on 720p CPU-resident frames.

According to the results, coordination improves end-to-end throughput by 1.59–2.05× over serial execution of the same modules, showing that the gains come from stage scheduling rather than changes to the model or compression policy. Although the individual components are not algorithmically novel, ViCoStream integrates these acceleration techniques into a unified system-level framework that makes high-throughput streaming video understanding practical. More importantly, it reveals how inference bottlenecks shift across processing stages as different parts of the pipeline are accelerated.

W2: It is not fully clear whether the reported 134 FPS represents sustained wall-clock throughput of the complete pipeline or an estimate based on stage-wise latency. ... the GPU stages share one A100.

We thank the reviewer for this question. We confirm that the 134-FPS entry is an estimate derived from stage-wise latency, 1000 × 16 / max(113, 119, 44, 42). To clarify more about ViCoStream's efficiency, we run the same 16-frame, 30%-retention configuration as a sustained wall-clock workload in the study.

CheckpointPixel budgetMeasured FPSElapsed (s)FramesStage estimateEstimate ÷ measured
Base 3B200,70493.3527.42,5601341.44×
Fine-tuned 3B200,70494.2027.22,5601341.42×
Fine-tuned 3B90,000159.3016.12,560——

According to the results, the original stage-based estimate is 1.42–1.44× higher than the measured throughput, consistent with shared-GPU contention and execution overhead that the idealized estimate does not capture. Nevertheless, ViCoStream achieves 93–94 FPS at the original 200,704-pixel budget and 159 FPS at 90,000 pixels. Furthermore, the serial controls in W1 show that coordinated execution improves throughput by 1.78–2.05× using the same model and compression settings. These results confirm substantial end-to-end acceleration of ViCoStream, despite the gap from the theoretical stage-based estimate.

W3: The current component studies make it difficult to isolate the benefit of coordination. ... A serial-versus-pipelined end-to-end control and an ablation of the adaptive training mask would help clarify how much each part contributes.

We thank the reviewer for these three requests.

For serial-versus-pipelined control: As provided in W1: the same modules run 1.59–2.05× faster under four-stage coordination than under serial execution, with preprocessing-only overlap accounting for an intermediate gain.

For Table 5(c) budget matching: The frame-level rows are calibrated to retain about 40% and 25% of visual tokens at s=0.25 and s=0.10 against the measured 42.4% and 27.1% for the chunk-level rows. In this study, we achieve the frame-level token dropping across adjacent frames rather than within a chunk so that the retained-token ratio differs by at most 2.6% between the two groups. Under this budget, chunk-level dropping leads at both thresholds: 70.68 against 68.80 at s=0.25 and 68.28 against 66.48 at s=0.10.

Input settingFrames per chunkacnsRetained tokensStreamingBench
Frame-level1160.2539.8%68.80
Frame-level1160.1025.1%66.48
Chunk-level840.2542.4%70.68
Chunk-level840.1027.1%68.28
Chunk-level880.2542.7%70.48
Chunk-level880.1027.1%67.68

Note that Qwen2.5-VL uses a default temporal_merge_size of 2, so the frame-level and chunk-level settings cover the same temporal span despite using different acn values. This ensures a fair comparison in terms of temporal coverage.

For adaptive training mask: The training mask is determined by acn and ur, which control how much historical context the model can access during training and inference. Actually, we train separate checkpoints for different acn/ur configurations, each using its corresponding attention mask. Varying acn and ur therefore constitutes an ablation of the training-mask configurations, rather than an inference-only attention modification. Moreover, applying an acn/ur setting at inference that differs from the one used in training would produce a train-eval mismatch, so in all experiments the mask is kept identical between the two. These configurations allow us to evaluate the effects of different attention scopes, as shown in the following table.

BackboneDropping thresholdFull-attention configurationBounded-attention configurationAccuracy change
Qwen2.5-VL-3B0.2569.9670.44+0.48
Qwen2.5-VL-3B0.1067.4068.32+0.92
Qwen2.5-VL-7B0.2573.2871.64−1.64
Qwen2.5-VL-7B0.1071.3669.71−1.65

According to the results, bounded attention improves accuracy by 0.48–0.92 points on 3B, while reducing it by 1.64–1.65 points on 7B. These models are trained with adaptive attention masks corresponding to their acn and ur settings, rather than applying bounded attention only at inference. Therefore, these results characterize the accuracy trade-offs of different training-mask configurations with matched inference attention, showing that the effect depends on the backbone.

W4: It is difficult to connect the headline efficiency and accuracy results at exactly the same operating point. ... Reporting accuracy and efficiency together for that configuration would make the central trade-off clearer.

We thank the reviewer for this suggestion. We now report both quantities for the exact configuration behind the headline number: c=16, 30% retention, acn=4, ur=64.

Pixel budgetMeasured FPSStreamingBenchOVO-Bench
90,00015966.5251.29
200,7049471.2052.13

The trade-off is now readable from a single row pair: the 90,000-pixel setting runs 65 FPS faster and gives up 4.68 points on StreamingBench, while the 200,704-pixel setting is the one behind the 134-FPS estimate.

Q1: A sustained wall-clock run of the complete pipeline under the (c=16), 30%-retention setting would be helpful. ... a serial or pipeline-disabled control would show whether the derived 134 FPS is achieved in practice.

We thank the reviewer for this request. We report the sustained run, the frame count, the elapsed time, and a serial control for the requested setting.

ConfigurationFramesElapsed (s)Serial FPSVP-overlap FPSFour-stage FPSSpeedup
Base 3B, 200,704 px, c=16, 30%2,56027.452.3869.0993.351.78×
Fine-tuned 3B, 200,704 px, c=16, 30%2,56027.2——94.20—
Fine-tuned 3B, 90,000 px, c=16, 30%2,56016.177.80114.13159.302.05×
Fine-tuned 3B, 90,000 px, c=8, s=0.251,28013.858.4480.6792.791.59×

For the configuration behind the 134-FPS estimate (c=16, 30% retention, 200,704 pixels), the base-3B variant completes 2,560 frames in 27.4 s at 93.35 FPS, against 52.38 FPS with the same modules in serial execution. The fine-tuned variant reaches 94.20 FPS. At 90,000 pixels the same pipeline completes 2,560 frames in 16.1 s at 159.30 FPS, against 77.80 FPS serially. The derived 134 FPS is not reached at the original budget, where the measured value is 93–94 FPS, but the coordinated pipeline does exceed 100 FPS at the lower visual budget.

Q2: Adding chunk size, streaming FPS, and query TTFT to Tables 3 and 4 ... Table 4 would also benefit from the measured token retention already reported for Table 3 ...

We thank the reviewer for this suggestion. Tables 3 and 4 use 8-frame chunks, while the headline configuration in Table 2 uses 16-frame chunks with fixed 30% token retention. We will clarify these settings in the revised captions.

Table 4 uses acn=4 and ur=32. Since ur does not affect token retention, the measured retention ratios from Table 3 also apply to Table 4:

Backbones_dropTokens droppedTokens retained
3B0.2557.6%42.4%
3B0.1072.9%27.1%
7B0.2548.0%52.0%
7B0.1070.6%29.4%

For the headline 16-frame configuration, measured complete TTFT is 121 ms at 90,000 pixels and 107 ms at 200,704 pixels, with 30-FPS input and queries every 10 seconds. The corresponding throughput and accuracy results are reported in W4.

Q3: Table 5(c) would be more informative if the frame-level and chunk-level rows matched the retained token count, visible temporal span, and LLM prefill latency. ... separate the effect of chunk-level dropping from the additional context or tokens retained by each configuration.

We thank the reviewer for this suggestion. As detailed in W3, we compare frame-level and chunk-level dropping under similar token budgets and the same temporal coverage. Their retention ratios differ by at most 2.6 percentage points. Chunk-level dropping improves StreamingBench accuracy by 1.88 and 1.80 points at the two thresholds, respectively. It also requires fewer prefill operations (4 versus 16), although prefill latency is not matched between the settings.

References

[1] Yao, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. ACM MM (2025).

[2] Zhang, et al. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv:2406.08085 (2024).

[3] Chen, et al. Streamingtom: Streaming token compression for efficient video understanding. arXiv:2510.18269 (2025).

Reviewer_KrhfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ViCoStream organizes streaming VideoLLM inference into visual preprocessing, chunk-wise visual encoding, token dropping, and LLM prefilling/decoding. It combines stage-level overlap with anchor-preserving temporal token dropping, local visual attention, query-driven chunk retrieval, and training masks aligned with restricted visibility. The evaluation uses Qwen2.5-VL-3B/7B and six streaming-oriented benchmarks, with additional component and cross-device analyses. The paper reports up to 134 FPS ingestion capacity and below-50-ms query-side TTFT on an A100, while retaining much of the full-history baseline's accuracy under moderate compression. Its principal contribution is the system-level organization and analysis of efficiency–accuracy trade-offs.

Critical strengths
  1. Useful analysis of interacting streaming bottlenecks. The distinction between ingestion throughput and query response time is well motivated (Section 3). Table 2 and Figures 4–5 examine how chunk size, token retention, attention locality, and retrieval scope affect different costs. This offers a useful framework for reasoning about why reducing LLM computation can expose front-end bottlenecks, which is consequential for practical streaming deployment.

  2. Broad evidence about the accuracy cost of bounded visual context. Tables 3–4 evaluate both 3B and 7B models across complementary streaming tasks; Table 5(a) controls token dropping when comparing attention scopes, and Table 7 tests alternative dropping modules inside the same pipeline. For example, the 3B setting with acn=4, ur=32, and s=0.25 obtains 70.68 versus 70.32 on StreamingBench and 45.16 versus 47.13 on OVO-Bench (Table 3). These results support useful, task-dependent trade-offs rather than a claim of uniform improvement.

Critical weaknesses
  1. The headline throughput and responsiveness are not established as one measured end-to-end operating point. In Table 2, c=16 at 30% retention gives stage times of 113/119/44/42 ms and 134 FPS, which matches 1000×16/max(113,119,44,42). On the shared GPU shown in Figure 6(c), that pipeline estimate requires overlap and resource-contention assumptions to be validated. The manuscript does not clarify whether stage timings were collected under concurrent execution or provide a completed-frame wall-clock throughput measurement. Moreover, TTFT starts when the query is submitted to the LLM (Section 3, lines 155–161), and Section 5.2, lines 381–384, explicitly separates last-chunk processing from this latency. Figure 2 also places visual prefilling and answer decoding in the same thread. Therefore, query load and visual-state freshness can affect the claimed responsiveness. The fastest c=16 setting is also not explicitly paired with an accuracy result; the detailed chunk-level comparisons use c=4 (Appendix A.6). A matched run reporting sustained ingestion, query-arrival-to-first-token latency, evidence freshness, and accuracy at the advertised settings would materially strengthen the central systems claim.

  2. Bounded attention does not yet establish the claimed bounds on historical storage and retrieval cost. Section 4, pages 5–6, scores the query against representative features of historical chunks and retrieves their KV caches, but does not specify a bounded archive, eviction policy, offloading/indexing scheme, or cache reconstruction procedure. If all N historical representatives are retained and scored as described, their storage and exact similarity computation grow with N; retaining historical KV states adds further storage. Figures 4–5 demonstrate latency behavior over finite histories, while Figure 1 reports full-history memory against chunk size rather than total ViCoStream memory against stream duration. This leaves the claim of arbitrarily long streams under fixed latency and memory budgets (Introduction, lines 76–90) insufficiently supported. The paper should account for the historical state and selection/data-movement costs, and clarify whether its bound applies to active attention, GPU memory, or total system resources.

Questions for authors
  1. End-to-end measurement: Were the Table 2 times measured with all stages concurrently active, and is the reported 134 FPS independently measured from completed frames or calculated from the maximum stage time? Please provide the achieved rate and query-arrival-to-first-token latency at c=16/30% retention, stating query frequency, answer length, retrieval budget, and the newest frame incorporated into each answer. Can the accuracy and below-50-ms TTFT claims be linked to this same configuration?

  2. Historical state: What exactly is retained for old chunks, where is it stored, and what limits its size? Does the reported query latency include similarity scoring, top-k selection, and retrieval or reconstruction of old KV states? A memory/latency trace over increasing history, or a precise qualification of the resource bound, would resolve this concern.

  3. Table 15 aggregation: Please clarify how OVO-Bench All is computed. The category weighting consistent with Table 13's displayed baseline values, (2×RT+BT+FAR)/4, yields 48.19, 48.92, and 48.79 for the three Table 15 baselines, rather than 44.66, 45.53, and 45.43; it reproduces ViCoStream's 45.30. Are different aggregation rules or evaluation subsets used, or are entries incorrect? This affects the stated 0.13-point gap to StreamingTOM in Appendix A.7.

4 comments

We thank the reviewer for the constructive assessment. We address the concerns regarding measured throughput, historical resource growth, query responsiveness, and benchmark aggregation below.

W1: The headline ... central systems claim.

We thank the reviewer for identifying this gap. The original 134 FPS was a stage-based estimate, computed as 1000 × 16 / max(113,119,44,42), rather than sustained wall-clock throughput. We have measured completed-frame throughput through the full four-stage pipeline and evaluated accuracy at the corresponding configurations. All experiments use a single A100, fine-tuned models, 16-frame chunks, 30% retention, acn=4, and ur=64 in this table.

BackbonePixel BudgetWall-clock FPS ↑StreamingBench ↑OVO-Bench ↑
Qwen2.5-VL-3B90,00015966.5251.29
Qwen2.5-VL-7B90,00015569.7250.61
Qwen2.5-VL-3B200,7049471.2052.13
Qwen2.5-VL-7B200,7049172.1251.77

The 90,000-pixel configuration exceeds 100 FPS in measured execution: the 3B model reaches 159 FPS with 66.52 on StreamingBench and 51.29 on OVO-Bench. At the original 200,704-pixel budget the same model achieves 94 FPS, trading throughput for accuracy (71.20 and 52.13). These results establish the previously missing throughput–accuracy operating without assuming ideal stage overlap.

We also compare the throughput of ViCoStream with ReKV, HERMES, ReKV + STC(P), and StreamingTOM on the same A100, using 200,704 pixels and 16-frame chunks in the following table.

MethodVP (ms)ViT (ms)Prefill (ms) ↓Query TTFT (ms) ↓Measured FPS ↑
Full-history119.3135.5727.8715.315.8
ReKV119.3137.6315.6361.831.2
HERMES131.7133.7307.261.935.7
ReKV + STC(P)122.5134.488.0158.264.0
StreamingTOM124.1150.997.4162.259.0
ViCoStream (Ours)1131194444.594.0

The strongest baseline, ReKV + STC(P), reaches 64.0 FPS against ViCoStream's 94.0 FPS (1.47×), and every baseline stays below 100 FPS in its native execution because a serial per-chunk path pays the sum of its stages rather than the maximum.

W2: Bounded attention ... system resources.

We thank the reviewer for raising this point. We agree that our implementation does not include specialized historical KV-cache management. Our algorithmic focus is to bound the attention context for each video update, avoiding attention over the entire accumulated history. This reduces per-update attention computation even though historical storage and exact retrieval scoring still grow with stream length. To quantify this benefit and its storage limitation, we evaluate Qwen2.5-VL-3B at 200,704 pixels over 300 chunks.

AttentionRetentionMean Prefill (ms) ↓First / Last 10 Chunks (ms)GPU Memory: First → Last (GB)
Full history50%295.4848.73 / 549.597.41 → 16.94
Bounded (acn=4)50%63.1546.79 / 80.367.41 → 15.44
Full history100%864.5787.35 / 1660.267.66 → 26.69
Bounded (acn=4)100%116.0880.22 / 150.107.66 → 23.63

These results demonstrate mean prefill speedups of 4.68× at 50% retention and 7.45× at full retention. This computational benefit does not require specialized KV-cache management, which is outside our main algorithmic focus. As a complementary study, we test fixed token budgets that keep recent KVs on the GPU and either evict or offload older entries to the CPU:

ConfigurationRetained KV tokensPeak GPU memoryQuery TTFT
No cache management72,8519.91 GB161.5 ms
Evict beyond 40k40,0008.82 GB141.9 ms
Evict beyond 20k20,0008.13 GB125.3 ms
Evict beyond 10k10,0007.79 GB111.7 ms
Offload beyond 20k20,0008.13 GB168.1 ms

These results illustrate two complementary approaches to managing historical KV storage. Eviction reduces GPU memory and query TTFT (161.5 → 111.7 ms) by limiting retained history, at the cost of discarding older evidence. CPU offloading instead preserves older KVs while reducing GPU memory, with a TTFT of 168.1 ms due to additional retrieval overhead. Both approaches manage historical storage independently of ViCoStream's bounded-attention computation, demonstrating that memory management can be incorporated without changing its attention-selection mechanism.

Q1: End-to-end measurement ... to this same configuration?

We thank the reviewer for this question. We confirm that the original 134 FPS is a stage-based estimate, computed from separately stage latencies rather than completed-frame measurements. To assess concurrent execution directly, we measure wall-clock throughput and query responsiveness on a single A100 using 16-frame chunks, 30% retention, acn=4, and ur=64. We process 240 chunks with queries every 10 seconds and a 32-token answer limit using visual-resolution budgets. The detailed results are shown below.

Pixel BudgetInput FPSWall-clock FPS ↑Complete TTFT (ms) ↓Service TTFT (ms) ↓Frontier Lag (s) ↓
90,0006059.7881390.20
90,0009087.88172820.11
90,000120113.88161860.13
200,7044039.5663630.40
200,7045554.453811320.13
200,70410077.713,3581310.08

In this table, Service TTFT measures the interval from model query submission to the first token. Complete TTFT starts at query arrival and includes queueing and retrieval. Frontier lag measures the timestamp gap between query arrival and the latest processed frame. At 90,000 pixels, ViCoStream achieves 113.88 FPS with 161-ms median complete TTFT under 120-FPS input. At 200,704 pixels, concurrent throughput reaches approximately 78 FPS.

Query frequency at 30-FPS input is also studied in the following table.

Pixel BudgetQuery IntervalWall-clock FPS ↑QueriesComplete TTFT (ms) ↓
90,0002 s29.864238
90,00010 s29.918121
200,7042 s23.2842866
200,70410 s29.898107

These results show the trade-off between accuracy, throughput, and query responsiveness. At 30-FPS input with queries every 2 seconds and 32-token answers, the 90,000-pixel setting sustains 29.86 to 30.09 FPS across three repetitions with median complete TTFT below 40 ms. The 200,704-pixel setting sustains 30-FPS ingestion when queries arrive every 10 seconds and improves evaluated StreamingBench/OVO-Bench accuracy from 66.52/51.29 to 71.20/52.13. At this higher budget, throughput is 94 FPS without queries and approximately 78 FPS under 100-FPS input with queries every 10 seconds, quantifying shared-resource contention beyond the idealized 134-FPS stage estimate.

Q2: Historical state: What exactly ... of old KV states?

We thank the reviewer for asking for this accounting. We retain historical visual KV caches, representative chunk features, and chunk/token-range metadata on the GPU. Historical KVs are directly gathered from the existing cache without re-encoding. Importantly, ViCoStream bounds the amount of historical context accessed during attention, rather than the total amount of historical information stored. This distinction reflects our algorithmic focus: reducing the computational cost of streaming attention while preserving access to historical evidence.

Regarding latency accounting, the isolated query profiler in the submission starts timing when the query enters the model, after context selection and cache assembly, so similarity scoring, top-k selection, and historical KV gathering fell outside its interval. The concurrent experiments in Q1 compensate for this gap: they report complete TTFT from query arrival, which includes retrieval, query processing, and queueing. A separate breakdown of the retrieval path measures 0.25 ms for similarity scoring and 0.26 ms for top-k selection and index mapping ( 0.51 ms in total), against the 39–132 ms service TTFT reported in Q1.

The experiment in W2 shows bounded attention reducing video-prefill computation while historical KV storage grows. Exact similarity scoring over historical representatives still scales with archive length, so archive management and active attention computation are complementary optimization problems. Hierarchical KV caching, CPU/GPU offloading, and paged KV allocation are established techniques that could be layered onto this design without changing its attention-selection mechanism. These results show that ViCoStream reduces attention computation while remaining compatible with standard KV-cache management techniques. Historical storage can be managed separately through eviction or offloading, without changing ViCoStream's attention mechanism.

Q3: Table 15 aggregation: Please clarify how OVO-Bench All is computed. ... in Appendix A.7.

We thank the reviewer for identifying this inconsistency. The three baseline All entries in Table 15 are inadvertently computed using the unweighted category mean. This is an aggregation error confined to those three entries, not a change in evaluation protocol. OVO-Bench contains twelve tasks (six RT, three BT, and three FAR), giving:

All = (2 × RT + BT + FAR) / 4.

MethodRTBTFARCorrected All
TimeChat-Online58.7840.0835.1248.19
Flash-VStream59.1042.7534.7248.92
StreamingTOM58.8842.1935.2248.79

The per-category results remain unchanged. Only the All column was calculated with incorrect weights. After correction, ViCoStream scores 45.30 compared with StreamingTOM's 48.79, giving a gap of 3.49% rather than 0.13.

The corrected results show a clearer accuracy–efficiency trade-off. ViCoStream achieves 94 FPS compared with 64 FPS for the strongest baseline, while scoring 3.49% lower than StreamingTOM on OVO-Bench. The accuracy gap is largest in backward tracing (4.83%), followed by real-time perception (3.14%) and forward active response (2.87%). This suggests that limiting historical attention affects tasks requiring older evidence more than those relying on recent observations. The correction changes the overall accuracy gap but not this task-level pattern.

References

[1] Yao, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. ACM MM (2025).

[2] Zhang, et al. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv:2406.08085 (2024).

[3] Chen, et al. Streamingtom: Streaming token compression for efficient video understanding. arXiv:2510.18269 (2025).

[4] Di, et al. Streaming video question-answering with in-context video KV-cache retrieval. ICLR (2025).

[5] Zhang, et al. HERMES: KV cache as hierarchical memory for efficient streaming video understanding. arXiv:2601.14724 (2026).

[6] Wang, et al. Accelerating streaming video large language models via hierarchical token compression. CVPR (2026).

Thanks for the detailed response. I will keep my original rating.

Reviewer_ym5gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViCoStream, a four-stage pipeline for streaming VideoLLMs. It drops redundant tokens, limits attention to recent chunks, and retrieves the most relevant chunks for each query. Qwen2.5-VL-3B/7B are fine-tuned with an attention mask that matches this design. Accuracy is mostly within 3 points of the full-history baselines at moderate dropping but up to 8 points lower at aggressive dropping. The authors claim that ViCoStream achieves 134 FPS and under 50 ms TTFT on an A100, with accuracy close to full-history baselines.

Critical strengths
  1. The multi-stage pipeline view is useful, and Table 2 shows that the bottleneck moves with chunk size. This analysis is the main contribution of the paper.
  2. Table 5b shows that CUDA-stream preprocessing is 7.7× to 11.8× faster than the CPU-based one. Since the front end becomes the bottleneck once the LLM cost is bounded, this speedup is useful for system design.
Critical weaknesses
  1. This paper does not report the throughput or TTFT of other streaming VideoLLMs. Since speed is the main claim, it is not clear whether ViCoStream is faster than existing methods, which makes the main contribution hard to evaluate.
  2. The 134 FPS in Table 2 uses 16 frames per chunk and 30% retention. It is computed rather than measured, and the accuracy of this setting is not reported. In Table 3, when about 30% of tokens are kept, StreamingBench accuracy drops by 2.0 to 5.0 points. This is the main reason for my rating.
Questions for authors
  1. Bounded memory is claimed, but only Fig. 1a reports memory, and it uses full-history attention. Where are old KVs stored, and how does GPU memory grow with stream length under bounded attention?
  2. It would be better to run several existing streaming VideoLLMs on the same A100 and compare their speed with ViCoStream.
  3. Can the actual throughput and accuracy of the 134 FPS setting be measured and reported?
5 comments

We thank the reviewer for recognizing the multi-stage bottleneck analysis and the preprocessing study. We respond to the two weaknesses and three questions individually below.

W1. This paper does not report the throughput or ... makes the main contribution hard to evaluate.

We thank the reviewer for this suggestion. We test ReKV, HERMES, ReKV + STC(P), and StreamingTOM on a single A100 with the Qwen2.5-VL-3B backbone, using the same 448×448 input, 16-frame chunks as Table 2. Each method is timed as it actually executes: the baselines process one chunk at a time, while ViCoStream runs the four stages concurrently. Throughput is completed frames divided by elapsed time over three repetitions, reported over the same chunk 80–100 window after warm-up, so the comparison uses one shared measurement interval.

MethodVP (ms)ViT (ms)Prefill (ms) ↓Query TTFT (ms) ↓Measured FPS ↑
Full-history119.3135.5727.8715.315.8
ReKV119.3137.6315.6361.831.2
HERMES131.7133.7307.261.935.7
ReKV + STC(P)122.5134.488.0158.264.0
StreamingTOM124.1150.997.4162.259.0
ViCoStream (Ours)1131194444.594.0

Baseline timings cover each method's model forward per chunk. Visual preprocessing and host-to-device transfer are outside their interval but inside ours, which makes the comparison conservative for ViCoStream. Query TTFT measures the interval from LLM query submission to the first generated token.

First, preprocessing and visual encoding costs are broadly comparable, while LLM prefill varies substantially. By restricting attention to a bounded number of historical chunks, ViCoStream makes per-chunk attention computation independent of accumulated video length, reducing prefill from 727.8 to 44 ms (16.5×) and raising measured throughput by 5.9× over full history.

Second, ViCoStream achieves higher measured throughput than existing methods. The fastest baseline, ReKV + STC(P), reaches 64.0 FPS, while ViCoStream achieves 94.0 FPS. This difference mainly comes from execution: existing methods process chunks sequentially, whereas ViCoStream overlaps different processing stages. These results compare the methods as implemented, rather than isolating the benefit of pipelining alone.

Third, ViCoStream achieves the highest throughput and lowest query TTFT among the compared methods. It runs at 94.0 FPS, 1.47× faster than the strongest baseline, ReKV + STC(P) (64.0 FPS). Its query TTFT is 44.5 ms, compared with 61.9 ms for HERMES and 158.2 ms for ReKV + STC(P). With a lower pixel budget of 90,000, ViCoStream further reaches 159 FPS, with corresponding accuracy reported in W2. These results show ViCoStream's speed advantage over existing streaming VideoLLMs.

W2: The 134 FPS in Table 2 uses 16 frames per chunk and 30% retention. ... It is computed rather than measured, and the accuracy of this setting is not reported. ... StreamingBench accuracy drops by 2.0 to 5.0 points.

We thank the reviewer for highlighting this issue. The original 134 FPS is a stage-based estimate rather than measured throughput. Moreover, Table 3 primarily uses a 90,000-pixel budget, while the efficiency analysis uses 200,704 pixels, preventing a direct accuracy–efficiency comparison. We conduct wall-clock throughput measurements and corresponding accuracy evaluations at both pixel budgets, using 16-frame chunks, 30% retention, acn=4, and ur=64 on a single A100.

BackbonePixel BudgetFPS ↑StreamingBench ↑OVO-Bench ↑
Qwen2.5-VL-3B90,00015966.5251.29
Qwen2.5-VL-7B90,00015569.7250.61
Qwen2.5-VL-3B200,7049471.2052.13
Qwen2.5-VL-7B200,7049172.1251.77

All configurations use fine-tuned models. We measure wall-clock inference throughput from CPU-resident video frames through the complete four-stage pipeline, including visual preprocessing, data transfers, and pipeline startup/drain. Accuracy is evaluated separately on benchmark videos using the corresponding model and pixel-budget settings.

The 90,000-pixel configuration exceeds 100 FPS in measured execution. At the original 200,704-pixel budget the 3B model measures 94 FPS, trading throughput for accuracy: 71.20 on StreamingBench and 52.13 on OVO-Bench against 66.52 and 51.29 at the lower budget. These results directly address the missing 16-frame accuracy evaluation and establish the measured throughput–accuracy trade-off.

Q1: Bounded memory is claimed ... Where are old KVs stored, and how does GPU memory grow with stream length under bounded attention?

We thank the reviewer for this question. In our implementation, historical visual KVs remain stored on the GPU. Bounded attention limits the historical keys accessed by each incoming chunk but does not evict stored KVs. Thus, per-chunk attention computation is bounded, whereas total historical KV storage grows with stream length.

We report measurements over one hour of video sampling using Qwen2.5-VL-3B with a 448²-pixel budget in the following Table.

AttentionRetentionMean Prefill (ms) ↓First / Last 10 Chunks (ms)GPU Memory: First → Last (GB)
Full history50%295.4848.73 / 549.597.41 → 16.94
Bounded (acn=4)50%63.1546.79 / 80.367.41 → 15.44
Full history100%864.5787.35 / 1660.267.66 → 26.69
Bounded (acn=4)100%116.0880.22 / 150.107.66 → 23.63

GPU memory denotes per-chunk peak allocated memory, including model weights and intermediate tensors.

At 50% retention, bounded attention reduces mean prefill latency by 4.68×, while GPU memory increases from 7.41 to 15.44 GB due to accumulated historical KVs. In our research scope, storage and computation are separate problems, and our contribution is on the algorithmic side: ViCoStream bounds both video-to-history attention and query-to-vision readout, which is what makes per-chunk computation independent of stream length and therefore enables sustained real-time inference. The memory held by historical KV states is a serving-infrastructure question with mature answers like hierarchical KV caching, CPU/GPU offloading, paged allocation, and eviction with an explicit retrieval horizon. These mechanisms attach to the retained cache without changing the attention-selection strategy that ViCoStream contributes. To explore the application in the speedup of cache management, we try to measure one such layer directly: holding the retained KV to a fixed budget keeps peak memory constant (9.91 → 7.79 GB across budgets from unbounded to 10k tokens) and lowers query TTFT from 161.5 ms to 111.7 ms. Therefore, we believe that our proposed ViCoStream complements the KV Cache management method in terms of computational efficiency.

Q2: It would be better to run several existing streaming VideoLLMs on the same A100 ... compare their speed with ViCoStream.

We thank the reviewer for this request. We provide the requested comparison in W1, including additional A100 efficiency comparisons with ReKV, HERMES, ReKV + STC(P), and StreamingTOM on the same setting as our Table 2 (c=16, 448×448 input, 200,704 pixels). This experiment covers prefill latency, query TTFT, and FPS. ViCoStream's end-to-end wall-clock throughput is 94 FPS at 200,704 pixels and 159 FPS at 90,000 pixels (W2).

We further summarize cross-backbone experiments to assess framework portability beyond Qwen2.5-VL, alongside the updated Qwen2.5-VL accuracy results in the following table.

BackboneVision EncoderLLMProfiled Max FPSStreamingBench (Baseline → Ours)OvO-All (Baseline → Ours)
Qwen2.5-VL-3BQwen2.5-VL ViTQwen2.5-3B134— → 71.2— → 52.13
Qwen2.5-VL-7BQwen2.5-VL ViTQwen2.5-7B130— → 72.12— → 51.77
LLaVA-OV-7BSigLIP-400MQwen2-7B9266.12 → 67.3247.48 → 45.84
MiniCPM-V-2.6-8BSigLIP-400MQwen2-7B9067.44 → 65.00—
VideoLLM-online-8BSigLIP-400MLLaMA-8B16835.99 → 34.7019.77 → 19.33

Max FPS comes from stage-level profiling at approximately 30% retention. Accuracy for the two Qwen2.5-VL rows is at fixed 30% retention, as in W2. The three additional backbones are evaluated at s=0.1.

Across three additional backbones, coordinated streaming changes StreamingBench accuracy by +1.20%, −2.44%, and −1.29%, respectively. These results demonstrate that the framework can be applied across different visual encoders and language backbones, which supports framework generalizability beyond Qwen2.5-VL.

Q3: Can the actual throughput and accuracy of the 134 FPS setting ... be measured and reported?

We thank the reviewer for this suggestion. As detailed in W2, we have measured the actual wall-clock throughput and corresponding accuracy of the originally advertised 16-frame, 30%-retention configuration on a single A100 GPU.

The original 134 FPS was a stage-based estimate at a 200,704-pixel budget. Under this setting Qwen2.5-VL-3B achieves 94 FPS in measured four-stage execution, with 71.20 on StreamingBench and 52.13 on OVO-Bench. The throughput rises to 159 FPS at a reduced 90,000-pixel budget, with accuracy of 66.52 and 51.29. These results directly illustrate the missing measured throughput and accuracy of the advertised configuration.

References

[1] Yao, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. ACM MM (2025).

[2] Zhang, et al. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv:2406.08085 (2024).

[3] Chen, et al. Streamingtom: Streaming token compression for efficient video understanding. arXiv:2510.18269 (2025).

[4] Di, et al. Streaming video question-answering with in-context video KV-cache retrieval. ICLR (2025).

[5] Zhang, et al. HERMES: KV cache as hierarchical memory for efficient streaming video understanding. arXiv:2601.14724 (2026).

[6] Wang, et al. Accelerating streaming video large language models via hierarchical token compression. CVPR (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.