acceptodds
Under review as a conference paper at ICLR 2027

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Abstract

Streaming video understanding is essential for real-time embodied and assistive systems, but processing open-ended streams with multimodal large language models (MLLMs) is computationally expensive. Existing methods reduce visual tokens or stored context, yet still apply full-depth prefill to admitted frames, causing repeated computation and depth-proportional KV-cache growth. We observe that shallow MLLM layers already provide effective signals for retrieving question-relevant evidence. Building on this observation, we propose **ShallowStream**, which decouples lightweight stream processing from selective full-depth answering. During streaming, ShallowStream processes frames only through shallow layers, using their KVs as a historical index with optional cluster compression. At query time, a text-only gate activates cross-layer token voting and diversity-aware retrieval when needed, after which only the selected history and recent context undergo full-depth prefill. ShallowStream reduces per-frame prefill and 20-second end-to-end latency by up to **52.1×** and **15.3×**, respectively, while improving average scores by approximately **1.7 points** over strong existing baselines. Our code is available at [https://anonymous.4open.science/r/ShallowStream/](https://anonymous.4open.science/r/ShallowStream/).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, MLLMs, Long-Context

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4zQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the asymmetry of streaming video understanding — frames arrive continuously while queries are sparse — and proposes scheduling model depth rather than tokens or cache size. The motivating observation (Figure 2) is that retrieval-relevant signal emerges within the first few language-model layers, so ShallowStream prefills every incoming video unit only through layers [0,P) (P=5 for Qwen3-VL-8B, P=4 for LLaVA-OneVision-7B) and keeps those shallow KVs plus input-level visual states as a full-history index, optionally compressing older units into running-average clusters once a memory budget B is exceeded. At query time, a ten-shot text-only query-logit gate first decides whether history is needed at all; if so, per-layer top-M attention over candidate visual tokens produces unit-level votes, a vote-ranked pool is narrowed by max-min diversity selection in a pooled-key descriptor space, selected units are expanded with one temporal neighbor, and only that evidence plus a short recent window is re-prefilled through the full stack to generate the answer — notably, the shallow KVs serve only as an index, since generation always re-prefills from layer 0. Across OVO-Bench and StreamingBench on two backbones, ShallowStream reports the best overall score in all four dataset-backbone settings (+1.7 points over the second-best same-backbone method) while claiming up to 52.1× lower per-frame prefill and 15.3× lower 20-second end-to-end latency, with bounded GPU memory and query latency essentially flat in history length. The depth-scheduling framing is genuinely novel and the systems evaluation is unusually thorough for this literature. My reservations are that the paper prunes only the language stack while leaving the vision encoder — which its own measurements suggest dominates the remaining per-frame cost — untouched and unexamined; that several quantities governing the memory and retrieval pipeline are never specified; that the evidence-retrieval cascade is under-explained and under-ablated; and that the per-category results are uneven in ways the paper does not discuss, including a gate whose recall is 44% on one of the two backbones.

Critical strengths
  1. Overall motivation for the paper is quite good, layer-wise diagnotic is an important direction to look for efficiency purposes.
  2. There is clear seperation of routing from retrieval and has been show in ablations.
Critical weaknesses

Weakness for the paper

  1. Why can’t video streaming be done through a vision encoder? Insetad of doing this in mmllm can this be done directly on clip level? Since clip model also has text encoder as well. Can the pruning be done on early layers of vision encoder itself? 

  2. How much is the budget b? How is it decided? 

  3. Section Optional Evidence Retrieval. Is very confusing, it’s not clear what has been done in this section. Why are there multiple levels of token pruning?  What is top 4K units? Where did 4 come from?

  4. Some of the results are confusing, in table 1 Real-Time Visual Perception in FPD, there seems to be no improvement? Why could be that?

Questions for authors

Overall some of the results are confusing in the paper ShallowStream does not improve: with LLaVA-OneVision it reaches 74.3, below SimpleStream (77.2), CausalMem (76.2), and WeaveTime (75.2), and with Qwen3-VL it reaches 75.3. Analysis is also weak in the paper.

4 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer 4zQ3

Thank you for your thoughtful and constructive review, and for recognizing the motivation for shallow indexing and the clear separation between routing and retrieval. Your insightful suggestions on alternative retrieval representations and your careful questions about the history budget and evidence-selection procedure have helped us sharpen the methodological explanation and strengthen the supporting experiments.

Below, we address each of your points with new experiments on vision-encoder retrieval and the history budget, alongside clarifications to the evidence-retrieval procedure.

W1: Can retrieval use a standalone vision encoder, with or without early-layer truncation?

Conclusion: Standalone vision–language encoders are a viable retrieval option, but yield somewhat lower downstream answer accuracy in our experiments. We therefore choose shallow VLM retrieval to balance retrieval quality and processing cost. We evaluated CLIP ViT-L/14@336 and SigLIP alongside shallow VLM retrieval, and found that cross-layer token voting gives higher downstream answer accuracy in our setting: 59.48%, compared with 57.47% for SigLIP and 56.61% for CLIP, with diversity selection and temporal expansion disabled (Figure 6). These comparisons motivate our choice of shallow VLM representations for query-relevant evidence selection.

Analysis: A plausible explanation is that retrieval benefits from both richer semantic knowledge and a more contextualized understanding of the question. CLIP and SigLIP encode images and text separately and rank them using global embedding similarity. Our retriever instead uses the VLM's language-model representations to contextualize the query and aggregate token-level relevance signals across shallow layers. This may help identify specific objects, actions, or relations required for answering, beyond broad image–text similarity.

Early-Layer Vision Truncation: This is a promising complementary direction. However, CLIP/SigLIP are trained to align their final image and text representations, so retrieval with intermediate visual features would require checking their alignment with the text encoder and potentially adapting it. Our current comparisons use the full pretrained encoders. ShallowStream retains training-free compatibility with existing VLMs and already achieves 10.29–12.81 ms/frame online processing with the full vision encoder included; combining shallow language-model indexing with an adapted lightweight vision encoder is a useful direction for further efficiency improvements.

W2: What is the history budget B, and how is it chosen?

Conclusion: The history budget B balances detailed historical coverage and GPU-memory efficiency: it determines how much history remains uncompressed before older units are consolidated into long-term clusters. Our preliminary budget sweep identified B = 64 units as a practical operating point for continuous streaming, and we keep it fixed in all LC-on experiments (Table R16). This corresponds to 128 sampled frames on Qwen3-VL.

Setup & Results: Fixing Qwen3-VL, 1 FPS, LC-on, the standard Gate, end-of-video queries, and a 32-frame prefill batch while varying only B (Table R16), larger B retains more detailed history and raises accuracy modestly from 47.80% to 48.80% at the cost of higher GPU memory, from 18.38 to 19.09 GiB. We therefore choose B = 64 for continuous streaming deployment and have added this setting to Implementation Details in Section 5.1.

Table R16: Effect of the detailed-history budget on GPU memory and accuracy.

B (historical units)Mean peak GPU memory (GiB)Overall accuracy (%)
6418.3847.80
9618.6748.40
12819.0948.80

W3: Why use multiple selection stages, and what does the 4 in “top 4K units” mean?

Conclusion: There is a misunderstanding here. Section 4.2 performs unit-level selection rather than multi-stage token pruning. Here, 4K denotes a 4 × K = 32 candidate pool, giving diversity selection headroom before narrowing to K = 8 units. We have expanded the description in Section 4.2 to clarify these steps and improve readability.

AuthorsOpenReview ↗

Response (2/2) to Reviewer 4zQ3

Procedure: Once the Gate activates retrieval, token-level signals assess the relevance of historical units; subsequent selection and temporal expansion operate on video units in three steps:

  1. Relevance screening (identify history relevant to the question): At each shallow layer, the top-M tokens with the highest attention from the question vote for their corresponding video units. The top 4K units by accumulated cross-layer votes form the candidate pool, aggregating fine-grained relevance signals across shallow layers and filtering irrelevant content.
  2. Diversity selection (reduce redundant evidence): Following the max-min selection principle in DivPrune, we iteratively select units that differ from those already selected within the relevance candidate pool, ultimately obtaining K complementary units. This reduces the budget spent on similar views and covers more relevant moments.
  3. Temporal expansion (preserve local temporal context): We add one preceding unit for each selected unit, providing action and scene context immediately before the selected moment and alleviating temporal discontinuities caused by sparse selection. The resulting evidence is merged with recent context, deduplicated, and ordered chronologically for answering with the full model.

Results: Figure 6 confirms that cross-layer token voting outperforms standalone vision–language encoders and pooled Q–K ranking, and diversity selection adds further gains. We clarified these three steps, the meaning of 4K, and the role of the factor of 4 in Optional Evidence Retrieval in Section 4.2.

W4 & Q1: How should the FPD results be interpreted?

Conclusion: Because individual subtasks have relatively small evaluation sets, their accuracy scores can be sensitive to a small number of predictions. ShallowStream improves FPD over both original backbones, gaining +2.0 points on OneVision and +7.0/+7.9 points on Qwen3-VL, alongside +13.1/+14.9 point overall Real-Time Perception gains (Table R17). FPD emphasizes short-horizon action anticipation, for which recent visual cues are particularly informative. Our FPD scores remain close to the strongest compared methods: on Qwen3-VL, ShallowStream matches SimpleStream at 75.3% and reaches 76.2% with LC-on. On OneVision, the 2.9-point difference between 74.3% and 77.2% corresponds to approximately 3 questions out of the 101-question FPD subset. This small question-count difference provides context for the subtask comparison, alongside ShallowStream's stronger aggregate Real-Time Perception results.

Results: Table R17 reproduces the relevant Table 1 scores to show FPD alongside aggregate Real-Time Visual Perception. On Qwen3-VL, ShallowStream matches SimpleStream (75.3%) and reaches 76.2% with LC-on; on OneVision, FPD relies on immediate short-horizon cues on a small evaluation subset, whereas ShallowStream's historical retrieval yields clear gains on aggregate Real-Time Visual Perception (75.4%/75.5% vs. 65.7%–71.0%).

Table R17: FPD and Real-Time Visual Perception averages on OVO-Bench (%, reproduced from Table 1 of the paper).

MethodOneVision FPDOneVision Real-Time avg.Qwen3-VL FPDQwen3-VL Real-Time avg.
Original Backbone72.362.368.366.1
SimpleStream77.271.075.380.2
CausalMem76.265.7——
WeaveTime75.268.8——
ShallowStream74.375.475.380.9
ShallowStream (LC-on)74.375.576.281.0

Dear Reviewer,

As the rebuttal deadline is only one day away, we would be grateful if you could kindly take a moment to review our responses to your questions and comments.

We have addressed your concerns through additional clarifications, experiments, and analyses, and highlighted the revised parts in the manuscript.

If you have any follow-up questions, we will do our best to address them. Thank you very much for your time and consideration.

Best regards,
The Authors

Dear Reviewer 4zQ3,

Sorry to follow up once more. With the author-reviewer discussion window closing very soon, we just wanted to check if you have had a chance to read our rebuttal and whether any points still need clarification. We sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_AuUWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ShallowStream, a training-free framework for streaming video understanding that allocates different computation budgets to continuous video processing and query-time answering. Incoming video units pass through only the shallow layers of an MLLM, whose key-value states form a historical index. When a question arrives, a text-only gate determines whether to retrieve historical evidence; token voting and diversity-aware selection then identify visual units for full-depth answering. Experiments on two MLLM backbones and two streaming benchmarks evaluate answer quality, latency, memory usage, and the contributions of individual components.

Critical strengths
  1. Well-motivated computation allocation. ShallowStream exploits the asymmetry between continuous video input and intermittent questions. Shallow indexing with selective full-depth answering offers a practical way to reduce repeated computation in streaming video understanding.

  2. Consistent improvements across evaluation settings. The method improves overall performance across two backbones and two benchmarks without task-specific training. With Qwen3-VL, it achieves 69.5 on OVO-Bench and 78.2 on StreamingBench, exceeding the respective strongest reported same-backbone baselines.

  3. Targeted ablations support the design choices.The layer-depth, routing, and retrieval ablations help explain the framework’s effectiveness. In particular, the results support shallow-layer retrieval and show additional benefits from token voting and diversity-aware evidence selection.

Critical weaknesses
  1. Comparisons use different sampling and context settings. The main tables mix baselines using 0.5, 1, or 2 FPS with SimpleStream’s fixed recent-frame window. These differences affect the available visual evidence and computational cost, complicating the comparison of accuracy and efficiency. Controlled comparisons with matched sampling rates and visual-token budgets would clarify the method’s advantages.

  2. The storage bound of long-history compression remains unclear. Equation (6) allows new clusters to accumulate under repeated scene changes, leaving total storage dependent on cluster count. Archived states are stored in host memory, while memory measurements exclude CPU capacity. The paper should specify how cluster growth is controlled and report host-memory usage over longer streams to substantiate its scalability claims.

  3. The reliability of text-only retrieval gating needs further analysis. Section 4.2 determines whether to retrieve history from the question text alone, although evidence requirements can also depend on the video context. Table 3 in Appendix C reports only 44% recall for the LLaVA gate on the synthetic calibration set, raising concerns about missed retrieval. Benchmark-level false-negative rates and their impact on answer accuracy should be analyzed.

Questions for authors

Q1. Controlled efficiency comparison. Could you report the raw latencies underlying the 52.1× and 15.3× speedups, together with the exact comparison settings? In particular, how does the final method compare with a full-depth indexing variant when sampling rate, visual-token count, context window, routing, retrieval, and implementation are held fixed?

Q2. Compression and storage guarantees. What values are used for the compression budget, similarity threshold, detailed-history window, and cluster retrieval budget? Is there a limit on the number of historical clusters? Please clarify the behavior under repeated scene changes and provide host-memory usage and cluster-count measurements as history grows.

Q3. Consequences of routing errors. Could you quantify the answer-quality loss caused by incorrectly skipping retrieval for each backbone? The 44% calibration recall reported for the LLaVA gate makes this particularly relevant. An analysis distinguishing unnecessary retrieval from missed necessary retrieval would help assess the robustness of the routing policy.

3 comments
AuthorsOpenReview ↗

Response (1/3) to Reviewer AuUW

Thank you for your comprehensive and constructive review, and for recognizing the motivation for allocating computation, the performance improvements across backbones, and the routing and retrieval ablations. We especially appreciate your insightful questions about comparison fairness and practical streaming costs, which have helped us make the evaluation more rigorous and the efficiency gains more explicit.

Below, we address your points with sampling-rate controls, a long-stream stress test, downstream Gate analysis, and clarified storage and latency metrics.

W1: Comparison principles under different sampling rates and context budgets

Conclusion: ShallowStream achieves competitive or stronger accuracy than HERMES and MuKV at much lower cost, under controlled 1-FPS evaluations (Table R8). Meanwhile, a higher sampling rate does not yield consistent gains for those baselines, supporting keeping each method's paper-reported settings in the main tables.

Results: Controlling HERMES and MuKV to 1 FPS (Table R8), HERMES improves on OneVision (59.49% on OVO-Bench, 75.51% on StreamingBench vs. ShallowStream's 62.2% and 75.5%) but drops noticeably on Qwen3-VL (53.46% and 73.31% vs. 69.5% and 78.2%), whereas MuKV drops on both benchmarks when raised from 0.5 to 1 FPS. Thus, uniform sampling rates do not consistently benefit every method and can penalize baselines outside their tuned regimes.

Analysis: Regarding visual-token budgets, methods retain history differently (e.g., HERMES retains 4K visual tokens, while ReKV stores all historical visual KVs in CPU memory), so forcing a single token cap would alter their native memory mechanisms; for HERMES, its 4K budget already outperforms 6K on OneVision. We keep original paper settings in the main tables and added the controlled 1-FPS comparisons to Appendix H.1 (Table 6).

Table R8: Overall accuracy of HERMES and MuKV before and after matching the sampling rate.

MethodBackboneBenchmarkOriginal sampling rate (FPS)Original score (%)Score at 1 FPS (%)
HERMESLLaVA-OneVision-7BOVO-Bench0.558.0759.49
HERMESQwen3-VL-8BOVO-Bench257.1953.46
HERMESLLaVA-OneVision-7BStreamingBench0.573.1575.51
HERMESQwen3-VL-8BStreamingBench276.5973.31
MuKVLLaVA-OneVision-7BOVO-Bench0.546.9146.45
MuKVLLaVA-OneVision-7BStreamingBench0.564.4061.40

W2 & Q2: Long-term storage growth, compression parameters, and host memory

Conclusion: Our current design reduces long-term memory growth rather than guaranteeing constant total memory, since new clusters can accumulate as visual content changes. We added a detailed ~114-minute continuous-stream evaluation in Figure 7 to demonstrate its practical benefits across memory, processing cost, and query latency. LC-on completes the stream with 19.04 GiB peak GPU memory and 781.88 MiB CPU archive storage across 631 clusters, while maintaining low online processing cost and stable query latency as history grows (Table R10; Figure 7). We will explore global cluster budgets and inter-cluster merging to further control memory over indefinitely long streams.

Compression Parameters: All LC-on experiments fix the detailed-history budget at B=64B = 64 units, the cosine-similarity threshold at τ=0.99\tau = 0.99, and the cluster retrieval budget at Kℓ≤4K_\ell \leq 4 (Table R11). The 64-unit detailed-history window corresponds to 128 frames on Qwen3-VL and 64 frames on LLaVA-OneVision. Recent and near-term context remain in detail, while older units are consolidated into long-term clusters.

Setup & Results: The long-stream experiment uses Qwen3-VL-8B on an RTX 4090 at 2 FPS, with a query every 60 seconds and outputs fixed to 16 tokens. At 1,024 sampled frames, compression reduces CPU visual-state storage from 504.00 to 81.03 MiB and peak GPU memory from 21.76 to 18.75 GiB (Table R10). LC-on then completes the full 114.3-minute stream, whereas LC-off exhausts GPU memory during minute 15. Figure 7(b) tracks CPU archive storage and cluster count throughout the stream, reaching 781.88 MiB and 631 clusters at completion.

Behavior under Scene Changes: Clustering is content-adaptive: a unit joins the latest cluster only when its shallow-feature descriptor has cosine similarity of at least 0.99 to the centroid; otherwise, it starts a new cluster. Our three-video ClipShots analysis illustrates this behavior: the video with 66 shot transitions produces 17 clusters, while the two videos without shot transitions produce 2 and 1, respectively, from the same 150 historical units (Table R9). Thus, changes in visual content lead to more retained historical representations.

AuthorsOpenReview ↗

Response (2/3) to Reviewer AuUW

Table R9: Content-adaptive long-history clustering on three ClipShots videos. Representation reduction is 1−(cluster count/150)1 - (\text{cluster count}/150); it measures the reduction in historical representation count.

VideoShot transitions in first 5 minTransitions/minHistorical units → clustersRepresentation reductionMaximum units per cluster
Gymnastics montage (frequent transitions)6613.2150 → 1788.7%35
Continuous cheerleading (no shot transitions)00.0150 → 298.7%143
School relay race (no shot transitions)00.0150 → 199.3%150

Storage Clarification: As corrected in Appendix D (Table 4), the CPU stores input-level visual states H0H^0, while shallow KVs and cluster KVs remain on the GPU. A fixed-size cluster representative refers to per-cluster storage, rather than a fixed number of clusters. The long-stream evaluation reports both CPU archive usage and GPU memory, making the storage cost across devices explicit.

Table R10: Storage costs of long-history compression. The five-video results are averages at 1,024 sampled frames. The long-video results are measured after LC-on completes the entire video.

Evaluation settingMethodPeak GPU memory (GiB)CPU H0H^0 (MiB)Cluster count
Five-video setting of Figure 4, 1,024 framesLC-off21.76504.000.0
Five-video setting of Figure 4, 1,024 framesLC-on18.7581.0318.8
Approximately 114 minutes of continuous videoLC-on19.04781.88631

Table R11: Compression and retrieval parameters for LC-on.

ParameterQwen3-VLLLaVA-OneVision
Compression budget / detailed-history window B64 units, 128 frames64 units, 64 frames
Cosine similarity threshold τ\tau0.990.99
Cluster retrieval budget KℓK_\ellUp to 4 cluster representativesUp to 4 cluster representatives

W3 & Q3: Reliability of the text-only Gate and consequences of routing errors

Conclusion: To address the concern about missed retrieval, we hold the retriever and answer model fixed and bypass the Gate by retrieving for every Backward question. Relative to this always-retrieve control, the Gate changes answer accuracy by +0.27 points on Qwen3-VL and −0.23 points on LLaVA-OneVision, while reducing retrieval calls by 51.7% and 81.5%, respectively (Table R12). Thus, its net impact on benchmark answer quality remains within 0.3 points. Moreover, unnecessary retrieval can introduce visual interference (Table R13), motivating our precision-first design.

This routing policy supports our central objective: competitive overall accuracy at substantially lower continuous-stream cost. ShallowStream maintains accuracy competitive with SOTA methods across both backbones and benchmarks (Tables 1–2). Combining shallow indexing with selective full-depth answering yields up to 52.1× faster per-frame processing and 15.3× lower accumulated computation per 20-second query interval (Figure 1; Table R14). Figure 4 shows reduced GPU-memory growth and low compute demand across query intervals, while Figure 7 demonstrates slow memory growth and stable processing cost and query latency over a ~114-minute stream.

Results: Because existing benchmarks lack per-question ground-truth labels indicating whether distant history is required, direct benchmark-level routing precision/recall is unavailable; instead, Table R12 compares no-retrieve, calibrated-Gate, and always-retrieve policies to assess the aggregate consequences of routing, examining both missed necessary retrieval (e.g., on Qwen3-VL ASI, the Gate captures the +12.84-point retrieval gain over no-retrieve) and unnecessary retrieval on EPM.

Furthermore, the concern about 44% recall implicitly assumes that more retrieval leads to better performance. On the QA-Ego4D-derived EPM subset (Table R13), where recent frames already contain relevant cues, indiscriminately enabling retrieval degrades reasoning by introducing multimodal visual interference:

  • Qwen3-VL: Always retrieving increases evidence coverage from 51.91% to 66.23%, yet answer accuracy drops markedly from 56.12% to 50.51% (a net loss of 5.61 percentage points).
  • LLaVA-OneVision: Evidence coverage increases from 37.86% to 61.47%, while accuracy decreases from 55.61% to 52.55% (a net loss of 3.06 percentage points).

ShallowStream targets training-free, low-latency streaming; for queries whose historical dependence lacks textual cues, we will explore lightweight joint visual–textual routing. The routing and EPM analyses are added as Tables 7 and 8 in Appendix H.2.

AuthorsOpenReview ↗

Response (3/3) to Reviewer AuUW

Table R12: Routing-policy comparison on OVO-Bench Backward under the same experimental settings. All accuracy scores are percentages.

BackboneRetrieval policyRetrieval rateEPMASIHLDBackward avg.
Qwen3-VLNo-retrieve (recent context only)0.00%53.5459.4655.3856.12
Qwen3-VLOriginal Gate48.34%52.572.349.558.1
Qwen3-VLAlways retrieve100.00%55.2270.9547.3157.83
LLaVA-OneVisionNo-retrieve (recent context only)0.00%51.5250.6844.0948.76
LLaVA-OneVisionOriginal Gate18.54%50.250.746.249.0
LLaVA-OneVisionAlways retrieve100.00%51.1851.3545.1649.23

Table R13: Evidence coverage and answer accuracy (%) on the QA-Ego4D-derived EPM subset, with questions asked near the end of the evidence interval. Corresponds to Table 8 in Appendix H.2 of the paper.

BackboneRetrieval policyEvidence coverageAnswer accuracy
Qwen3-VLNo-retrieve51.9156.12
Qwen3-VLAlways retrieve66.2350.51
LLaVA-OneVisionNo-retrieve37.8655.61
LLaVA-OneVisionAlways retrieve61.4752.55

Q1: Absolute latencies, comparison settings, and the full-depth control

Conclusion: We have added detailed absolute-latency measurements and a controlled full-depth indexing comparison to the revised manuscript (Appendix H.3; Section 5.4, Figure 7, and Appendix I). The 52.1× and 15.3× speedups correspond to 10.29 vs. 536.10 ms/frame in online processing and 1.19 vs. 18.22 s in accumulated computation over a 20-second query interval (Table R14). Under strictly fixed settings, P = 5 reduces per-frame cost by 58.9% over full-depth indexing at P = 36, cutting cost from 24.43 to 10.05 ms/frame while avoiding the minute-2 OOM (Table R15).

Setup: Table R14 uses Qwen3-VL-8B on one NVIDIA RTX 5090 over the same five long OVO-Bench Backward videos at 1 FPS, with visible output fixed to 16 tokens and video decoding excluded from timing (ShallowStream uses P = 5, LC-off, and the final Gate and retrieval configurations). The 20-second metric accumulates stream processing and complete query computation within that interval, while user-perceived TTFT is reported separately in Figure 7(c).

Depth Controls: Figure 2 of the original paper already reports retrieval quality and prefill cost from shallow to full-depth processing for both backbones, and Figure 9 and Table 5 in Appendix E show that deeper processing increases per-frame computation while P = 5 maintains a better Backward average accuracy. In the new Figure 7, we fix the sampling rate, visual-token settings, context and memory budgets, routing sequence, and retrieval rules within the same implementation, changing only the indexing depth from P = 5 to full P = 36 layers (Table R15). Under this controlled comparison, P = 5 reduces per-frame cost by 58.9% (10.05 vs. 24.43 ms/frame), and the full-depth variant runs out of GPU memory during minute 2 whereas LC-on completes the ~114-minute video with stable TTFT (Figures 7(c)–7(d), Appendix I, and Table 10).

Table R14: Absolute costs underlying the speedups in Figure 1.

MethodPer-frame online processing (ms)Accumulated computation over a 20-second interval (s)
ShallowStream10.291.19
OASIS536.1018.22

Table R15: Depth controls with all other settings held fixed within each comparison. Costs are in ms/frame; reductions are relative to the deeper setting in each row. D.S.: Deeper Setting.

ComparisonP = 5D.S.Cost of D.S.Cost Reduction
Appendix E, RTX 5090, final Gate and retrieval configurations10.72P = 1915.5331.0%
New stress test, RTX 4090, identical first-minute input10.05P = 36 (full depth)24.4358.9%
Reviewer_h1NaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This approach offloads the computational burden of processing continuous video streams to the shallower layers of a language model, while simultaneously building a searchable historical index. When a query arises, the system selects a small number of relevant video segments and uses the full-scale model to perform inference and generate an answer. This primarily addresses the high computational costs associated with continuous video stream understanding, while also improving the balance between historical retrieval and the interpretation of the current frame.

Critical strengths

I believe there are three key advantages:

  • Direct reduction of historical computation costs. Compressing data after full computation cannot recover the costs already incurred; shallow processing, by contrast, reduces the cost per frame at the source.
  • Decoupling retrieval from reasoning. For the bulk of historical data, only representations sufficient for retrieval need to be stored; full-scale reasoning is reserved exclusively for a small, selected subset of evidence.
  • Simultaneous handling of "whether to retrieve" and "what to retrieve." A gating mechanism filters out irrelevant historical input, token-based voting pinpoints specific local evidence, and diversity-focused strategies minimize redundant content.
Critical weaknesses
  1. The text-only gate may prevent retrieval of necessary historical evidence. The gate cannot inspect the video to determine whether recent context is sufficient. On the independent synthetic calibration set, the LLaVA gate achieves 97.78% precision but only 44.00% recall, missing over half of the questions labeled as requiring history. The paper does not quantify the downstream impact of routing errors on the evaluation benchmarks. This weakens confidence that the system reliably accesses historical evidence when needed.

  2. Long-history compression lacks targeted evaluation of evidence preservation. Each historical cluster retains averaged shallow states and a single visual representative, which may lose brief events, small objects, or information needed for exact counting. Although aggregate scores change little with compression enabled, these averages do not establish preservation of such details.

  3. Bounded total memory is not established for indefinitely long streams. Fixed-size storage per cluster does not guarantee bounded total storage when new clusters can continue to accumulate. The method does not specify a global cluster limit or a policy that guarantees bounded memory. Moreover, the memory evaluation excludes CPU archive capacity. This leaves a central scalability claim insufficiently supported.

  4. The results do not consistently demonstrate improved historical understanding. On Qwen3-VL-8B, EPM accuracy is 52.5 versus 62.0 for OASIS; on LLaVA-OneVision-7B, the Backward Tracing average also falls below some baselines. These results do not invalidate the overall quality–efficiency trade-off, but they limit claims of stronger historical evidence use. The paper lacks an analysis separating routing, retrieval, and generation errors to explain these regressions.

  5. The efficiency evaluation provides limited support for broad deployment claims. The main measurements use five videos, one GPU, 1 FPS, and short outputs, excluding video decoding. Higher frame rates, frequent queries, longer answers, and substantially longer streams remain insufficiently evaluated. Furthermore, interval-level end-to-end gains combine continuous processing and query computation, so they do not directly establish comparable reductions in user-perceived response latency. These limitations are consequential because efficient continuous operation is the paper’s principal contribution.

Questions for authors

How does the text-only gate handle questions whose historical requirements depend on the actual video content? The LLaVA gate achieves only 44% recall on the synthetic calibration set. Could you report routing precision and recall on the evaluation benchmarks, and quantify how often incorrect recent-only routing causes an otherwise answerable question to fail? An oracle-routing comparison would help separate routing failures from retrieval and generation errors.

Does long-history compression preserve brief but decisive evidence? Could you evaluate compression specifically on short-lived events, small objects, and exact counting, rather than relying primarily on aggregate accuracy? Please also clarify whether evidence discarded by a cluster representative can ever be recovered.

What guarantees bounded total storage as the stream grows indefinitely? The method provides fixed-size representations per cluster, but the number of clusters may continue increasing. Is there a global cluster budget, eviction policy, or further consolidation mechanism? Could you report CPU archive usage alongside GPU memory over substantially longer streams, including videos with frequent scene changes?

What causes the regressions in historical understanding? Qwen3-VL-8B obtains 52.5 on EPM versus 62.0 for OASIS, while the LLaVA version’s Backward Tracing average falls below some baselines. Could you determine whether these regressions primarily arise from gate decisions, evidence selection, compression, or answer generation?

How robust are the efficiency gains across operating conditions, and how much do they improve query response latency? Could you separately report continuous processing cost, time to first token, and complete response latency under matched output lengths, including P50/P95 latency? Additional measurements across higher frame rates, shorter query intervals, and longer streams would clarify when retrieval and repeated full-depth prefill offset the shallow-processing savings. Please also distinguish interval-level accumulated computation from user-perceived response latency.

Could you strengthen comparisons with relevant CVPR 2026 methods, particularly FluxMem, MuKV, and StreamReady? FluxMem provides a training-free adaptive compression baseline, MuKV addresses multi-grained historical retrieval, and StreamReady reports strong Backward Tracing results. These methods are absent from the main experimental comparisons, although MuKV and StreamReady are cited. Could you compare the most applicable methods under matched backbones, causal access, sampling rates, and memory budgets, while distinguishing training requirements? Reporting both accuracy and total processing/storage costs would help establish whether ShallowStream’s advantage remains against these alternatives.

6 comments
AuthorsOpenReview ↗

Response (1/4) to Reviewer h1Na

Thank you for your thorough, insightful, and constructive review. We particularly appreciate your careful examination of Gate reliability, long-history compression, and continuous-operation efficiency, as well as your helpful suggestions for relevant baselines. Your comments have helped us strengthen the empirical evaluation and clarify the design choices behind ShallowStream.

Below, we address each of your points with downstream routing analysis, long-video storage and latency measurements, and MuKV/FluxMem/StreamReady comparisons.

W1 & Q1: Reliability of the text-only Gate and the downstream impact of missed retrieval

Conclusion: Visually grounded routing in methods such as OASIS requires multimodal inference to assess whether recent context is sufficient. To make continuous streaming practical, we instead use a lightweight, training-free text-only Gate and deliberately prioritize retrieval precision, reducing unnecessary computation and visual interference (Appendix C). This design forms part of a system that maintains accuracy competitive with SOTA methods on OVO-Bench and StreamingBench (Tables 1–2) while achieving 52.1× faster per-frame online processing than OASIS (10.29 vs. 536.10 ms/frame, Figure 1; Table R6). We address the four parts of Q1 below.

  1. How are questions handled when historical requirements depend on the video? Table R1 shows that even on OVO-Bench Backward, which is designed to evaluate historical understanding, retrieving for every question does not consistently improve accuracy over our Gate. The Gate achieves 58.1% vs. 57.83% on Qwen3-VL, while substantially reducing retrieval calls. These results suggest that video-dependent retrieval requirements have limited impact on aggregate performance on the evaluated benchmarks. In more complex scenarios, routing may benefit from jointly considering the query and visual history; we will explore lightweight joint visual–textual gating in future work.

  2. What does 44% calibration recall mean, and what are benchmark precision/recall? The 44% recall accompanies 97.78% retrieval precision for LLaVA-OneVision on the independent synthetic set. It follows our deliberate precision-first calibration (Appendix C). OVO-Bench and StreamingBench do not provide per-question labels of whether recent context is sufficient, so routing precision/recall under this definition cannot be computed directly from their answer labels. We therefore report benchmark answer accuracy and retrieval frequency under controlled routing policies in Table R1.

  3. How often does incorrect recent-only routing cause an otherwise answerable question to fail? We examine the downstream consequences using both Tables R1 and R2. In Table R1, the Gate stays within 0.3 points of always retrieving on both backbones while cutting 51.7% and 81.5% of retrieval calls. Table R2 further shows why more retrieval does not necessarily improve answer accuracy: on the QA-Ego4D-derived EPM subset, always retrieving increases evidence coverage from 51.91% to 66.23% on Qwen3-VL, yet accuracy drops from 56.12% to 50.51%; LLaVA-OneVision exhibits the same pattern, with a 3.06-point accuracy drop despite higher coverage. This indicates that the evaluated VLMs are susceptible to interference from additional visual context, even when retrieval brings in more relevant evidence. These results motivate our precision-first Gate: selectively triggering retrieval reduces unnecessary computation and visual interference while preserving overall answer quality.

  4. Does the comparison isolate routing errors through an oracle? OVO-Bench Backward is designed to require historical evidence, making always-retrieve a task-based proxy for oracle routing. In Table R1, this policy activates retrieval for every question, eliminating skipped retrieval while keeping the retriever and answer model fixed. Compared with this reference, our Gate achieves 58.1% vs. 57.83% on Qwen3-VL and 49.0% vs. 49.23% on LLaVA-OneVision. Thus, bypassing the Gate to ensure historical retrieval for all Backward questions yields no consistent accuracy gain, supporting the effectiveness of our lightweight routing policy.

AuthorsOpenReview ↗

Response (2/4) to Reviewer h1Na

Table R1: Routing-policy comparison on OVO-Bench Backward under the same experimental settings. All accuracy scores are percentages.

BackboneRetrieval policyRetrieval rateEPMASIHLDBackward avg.
Qwen3-VLNo-retrieve (recent context only)0.00%53.5459.4655.3856.12
Qwen3-VLOriginal Gate48.34%52.572.349.558.1
Qwen3-VLAlways retrieve100.00%55.2270.9547.3157.83
LLaVA-OneVisionNo-retrieve (recent context only)0.00%51.5250.6844.0948.76
LLaVA-OneVisionOriginal Gate18.54%50.250.746.249.0
LLaVA-OneVisionAlways retrieve100.00%51.1851.3545.1649.23

Table R2: Evidence coverage and answer accuracy (%) on the QA-Ego4D-derived EPM subset, with questions asked near the end of the evidence interval. Corresponds to Table 8 in Appendix H.2 of the paper.

BackboneRetrieval policyEvidence coverageAnswer accuracy
Qwen3-VLNo-retrieve51.9156.12
Qwen3-VLAlways retrieve66.2350.51
LLaVA-OneVisionNo-retrieve37.8655.61
LLaVA-OneVisionAlways retrieve61.4752.55

W2 & Q2: What does long-history compression preserve, and can discarded details be recovered?

Conclusion: Long-history compression uses a high cosine-similarity threshold of τ=0.99\tau = 0.99, merging a historical unit into the latest cluster only when its shallow-feature descriptor closely matches the cluster centroid. Our additional analysis of three ClipShots videos shows that ShallowStream allocates more clusters to videos with more frequent scene transitions: the transition-rich gymnastics video produces 17 clusters, compared with 2 and 1 for the two videos without shot transitions (Table R3). This supports our design of compressing repetitive history while allocating separate representations to changes in visual content.

Setup & Results: We process the first 300 seconds of each video, followed by the immediately subsequent 128 seconds of real video, allowing the first 150 units to enter long-term clusters naturally. All three cases use Qwen3-VL-8B-Instruct, BF16, FlashAttention 2, a question-only Gate, and pruning boundary P = 5. Shot-transition counts refer to the initial 300 seconds. The transition-rich video retains more historical representations and has a smaller maximum cluster, whereas the continuous scenes permit greater consolidation of repetitive content.

Table R3: Content-adaptive long-history clustering on three ClipShots videos. Representation reduction is 1−(cluster count/150)1 - (\text{cluster count}/150); it measures the reduction in historical representation count.

VideoShot transitions in first 5 minTransitions/minHistorical units → clustersRepresentation reductionMaximum units per cluster
Gymnastics montage (frequent transitions)6613.2150 → 1788.7%35
Continuous cheerleading (no shot transitions)00.0150 → 298.7%143
School relay race (no shot transitions)00.0150 → 199.3%150

Clarification: As specified in Optional Long-History Compression in Section 4.1, clustering operates over temporally adjacent units: a unit starts a new cluster whenever its similarity to the latest centroid falls below τ\tau, allowing the representation count to adapt to visual changes. Compression applies only to older history; recent context and near-term history remain in detail. Each long-term cluster retains averaged shallow KVs for indexing and one input-level visual representative for full-depth answering, preserving event-level summaries and scene semantics. Retrieval accesses this stored representative rather than reconstructing every original member. LC-off retains the full historical states when exact access to all original units is required.

AuthorsOpenReview ↗

Response (3/4) to Reviewer h1Na

W3, W5, Q3 & Q5: Long-term storage growth, continuous processing cost, and query response latency

Conclusion: Our current design reduces long-term memory growth rather than guaranteeing constant total memory, since new clusters can accumulate as visual content changes. We added a detailed ~114-minute continuous-stream evaluation in Figure 7 to demonstrate its practical benefits across memory, processing cost, and query latency. LC-on completes the stream with 19.04 GiB peak GPU memory and 781.88 MiB CPU archive storage, while maintaining 12.81 ms/frame online processing and stable query latency as history grows (Tables R4–R5). Mean TTFT is 0.30 s without retrieval and 1.23 s with retrieval, directly measuring user-perceived response time separately from accumulated computation. We will explore global cluster budgets and inter-cluster merging to further control memory over indefinitely long streams.

Setup: All methods use Qwen3-VL-8B on an RTX 4090 at 2 FPS, with a query after every 60 seconds of processed video and history retained continuously. ShallowStream uses a fixed routing workload that activates retrieval for approximately half of the queries while retaining Gate computation. Query timing starts after question-independent stream processing finishes and includes gating, retrieval, processing of the answer input, and generation.

Results:

  • Storage & Online Processing (Table R4): LC-on completes the entire 114.3-minute video with GPU memory (19.04 GiB) slightly above FluxMem's fixed-capacity memory (18.24 GiB), while maintaining a lower per-frame online processing cost (12.81 vs. 16.78 ms/frame). LC-off and OASIS run out of GPU memory at minutes 15 and 13, respectively. Figure 7(b) tracks CPU H0H^0 storage and cluster count, reaching 781.88 MiB and 631 clusters at the end of the video. We have corrected Appendix D to reflect that the CPU stores input-level visual states H0H^0 while shallow and cluster KVs remain on GPU, and clarified our paper claims as reducing storage growth over long streams (with a global budget and inter-cluster merging left for future extension).
  • Query Latency (Table R5): LC-on responds rapidly when retrieval is unnecessary (0.30 s mean TTFT, 0.75 s complete response) and maintains tight P50/P95 bounds when retrieving (1.23 s / 1.27 s TTFT; 1.70 s / 1.75 s complete response), remaining flat as the video grows longer (Figure 7). Added time-series curves to Section 5.4 and Figure 7 and full P50/P95 statistics to Table 10 in Appendix I.

Table R4: Storage and online processing costs in the long-stream stress test. Per-frame times include visual encoding.

MethodVideo duration completed (min)Peak GPU memory (GiB)Online processing (ms/frame)
LC-on114.319.0412.81
LC-off14.022.8612.73
FluxMem (Qwen3-VL adaptation)114.318.2416.78
OASIS13.022.98331.99

Table R5: Query latency in the long-stream stress test (seconds), matching Table 10 in Appendix I. Outputs are fixed to 16 tokens; statistics cover successful queries.

MethodRetrieval pathMean TTFTTTFT P50TTFT P95Mean complete responseComplete response P50Complete response P95
LC-onRetrieve1.231.231.271.701.701.75
LC-onNo-retrieve0.300.300.310.750.760.77
LC-offRetrieve1.221.151.421.691.621.89
LC-offNo-retrieve0.340.330.420.800.780.87
FluxMem (Qwen3-VL adaptation)—1.071.061.281.541.531.76
OASIS—5.895.747.276.436.287.79

W4 & Q4: Which components explain the differences across historical-understanding subtasks?

Conclusion: The subtask differences should be viewed in the context of different accuracy–computation trade-offs. OASIS uses recent visual context and long-term summaries to guide retrieval, investing substantially more computation in preparing and accessing historical evidence. This richer processing can benefit individual historical-understanding subtasks. ShallowStream instead targets overall accuracy at low continuous-stream cost: on Qwen3-VL, it achieves 69.5% overall on OVO-Bench versus 67.7% for OASIS (Table 1), while processing frames 52.1× faster (10.29 vs. 536.10 ms/frame, Table R6). Our objective is to make continuous video understanding practical through this overall accuracy–efficiency balance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.