- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets the asymmetry of streaming video understanding — frames arrive continuously while queries are sparse — and proposes scheduling model depth rather than tokens or cache size. The motivating observation (Figure 2) is that retrieval-relevant signal emerges within the first few language-model layers, so ShallowStream prefills every incoming video unit only through layers [0,P) (P=5 for Qwen3-VL-8B, P=4 for LLaVA-OneVision-7B) and keeps those shallow KVs plus input-level visual states as a full-history index, optionally compressing older units into running-average clusters once a memory budget B is exceeded. At query time, a ten-shot text-only query-logit gate first decides whether history is needed at all; if so, per-layer top-M attention over candidate visual tokens produces unit-level votes, a vote-ranked pool is narrowed by max-min diversity selection in a pooled-key descriptor space, selected units are expanded with one temporal neighbor, and only that evidence plus a short recent window is re-prefilled through the full stack to generate the answer — notably, the shallow KVs serve only as an index, since generation always re-prefills from layer 0. Across OVO-Bench and StreamingBench on two backbones, ShallowStream reports the best overall score in all four dataset-backbone settings (+1.7 points over the second-best same-backbone method) while claiming up to 52.1× lower per-frame prefill and 15.3× lower 20-second end-to-end latency, with bounded GPU memory and query latency essentially flat in history length. The depth-scheduling framing is genuinely novel and the systems evaluation is unusually thorough for this literature. My reservations are that the paper prunes only the language stack while leaving the vision encoder — which its own measurements suggest dominates the remaining per-frame cost — untouched and unexamined; that several quantities governing the memory and retrieval pipeline are never specified; that the evidence-retrieval cascade is under-explained and under-ablated; and that the per-category results are uneven in ways the paper does not discuss, including a gate whose recall is 44% on one of the two backbones.
Critical strengths
- Overall motivation for the paper is quite good, layer-wise diagnotic is an important direction to look for efficiency purposes.
- There is clear seperation of routing from retrieval and has been show in ablations.
Critical weaknesses
Weakness for the paper
-
Why can’t video streaming be done through a vision encoder? Insetad of doing this in mmllm can this be done directly on clip level? Since clip model also has text encoder as well. Can the pruning be done on early layers of vision encoder itself?
-
How much is the budget b? How is it decided?
-
Section Optional Evidence Retrieval. Is very confusing, it’s not clear what has been done in this section. Why are there multiple levels of token pruning? What is top 4K units? Where did 4 come from?
-
Some of the results are confusing, in table 1 Real-Time Visual Perception in FPD, there seems to be no improvement? Why could be that?
Questions for authors
Overall some of the results are confusing in the paper ShallowStream does not improve: with LLaVA-OneVision it reaches 74.3, below SimpleStream (77.2), CausalMem (76.2), and WeaveTime (75.2), and with Qwen3-VL it reaches 75.3. Analysis is also weak in the paper.
4 comments
Response (2/2) to Reviewer 4zQ3
Procedure: Once the Gate activates retrieval, token-level signals assess the relevance of historical units; subsequent selection and temporal expansion operate on video units in three steps:
- Relevance screening (identify history relevant to the question): At each shallow layer, the top-M tokens with the highest attention from the question vote for their corresponding video units. The top 4K units by accumulated cross-layer votes form the candidate pool, aggregating fine-grained relevance signals across shallow layers and filtering irrelevant content.
- Diversity selection (reduce redundant evidence): Following the max-min selection principle in DivPrune, we iteratively select units that differ from those already selected within the relevance candidate pool, ultimately obtaining K complementary units. This reduces the budget spent on similar views and covers more relevant moments.
- Temporal expansion (preserve local temporal context): We add one preceding unit for each selected unit, providing action and scene context immediately before the selected moment and alleviating temporal discontinuities caused by sparse selection. The resulting evidence is merged with recent context, deduplicated, and ordered chronologically for answering with the full model.
Results: Figure 6 confirms that cross-layer token voting outperforms standalone vision–language encoders and pooled Q–K ranking, and diversity selection adds further gains. We clarified these three steps, the meaning of 4K, and the role of the factor of 4 in Optional Evidence Retrieval in Section 4.2.
W4 & Q1: How should the FPD results be interpreted?
Conclusion: Because individual subtasks have relatively small evaluation sets, their accuracy scores can be sensitive to a small number of predictions. ShallowStream improves FPD over both original backbones, gaining +2.0 points on OneVision and +7.0/+7.9 points on Qwen3-VL, alongside +13.1/+14.9 point overall Real-Time Perception gains (Table R17). FPD emphasizes short-horizon action anticipation, for which recent visual cues are particularly informative. Our FPD scores remain close to the strongest compared methods: on Qwen3-VL, ShallowStream matches SimpleStream at 75.3% and reaches 76.2% with LC-on. On OneVision, the 2.9-point difference between 74.3% and 77.2% corresponds to approximately 3 questions out of the 101-question FPD subset. This small question-count difference provides context for the subtask comparison, alongside ShallowStream's stronger aggregate Real-Time Perception results.
Results: Table R17 reproduces the relevant Table 1 scores to show FPD alongside aggregate Real-Time Visual Perception. On Qwen3-VL, ShallowStream matches SimpleStream (75.3%) and reaches 76.2% with LC-on; on OneVision, FPD relies on immediate short-horizon cues on a small evaluation subset, whereas ShallowStream's historical retrieval yields clear gains on aggregate Real-Time Visual Perception (75.4%/75.5% vs. 65.7%–71.0%).
Table R17: FPD and Real-Time Visual Perception averages on OVO-Bench (%, reproduced from Table 1 of the paper).
| Method | OneVision FPD | OneVision Real-Time avg. | Qwen3-VL FPD | Qwen3-VL Real-Time avg. |
|---|---|---|---|---|
| Original Backbone | 72.3 | 62.3 | 68.3 | 66.1 |
| SimpleStream | 77.2 | 71.0 | 75.3 | 80.2 |
| CausalMem | 76.2 | 65.7 | — | — |
| WeaveTime | 75.2 | 68.8 | — | — |
| ShallowStream | 74.3 | 75.4 | 75.3 | 80.9 |
| ShallowStream (LC-on) | 74.3 | 75.5 | 76.2 | 81.0 |
Dear Reviewer,
As the rebuttal deadline is only one day away, we would be grateful if you could kindly take a moment to review our responses to your questions and comments.
We have addressed your concerns through additional clarifications, experiments, and analyses, and highlighted the revised parts in the manuscript.
If you have any follow-up questions, we will do our best to address them. Thank you very much for your time and consideration.
Best regards,
The Authors
Dear Reviewer 4zQ3,
Sorry to follow up once more. With the author-reviewer discussion window closing very soon, we just wanted to check if you have had a chance to read our rebuttal and whether any points still need clarification. We sincerely appreciate your time and consideration.
Best regards,
The Authors
Response (1/2) to Reviewer 4zQ3
Thank you for your thoughtful and constructive review, and for recognizing the motivation for shallow indexing and the clear separation between routing and retrieval. Your insightful suggestions on alternative retrieval representations and your careful questions about the history budget and evidence-selection procedure have helped us sharpen the methodological explanation and strengthen the supporting experiments.
Below, we address each of your points with new experiments on vision-encoder retrieval and the history budget, alongside clarifications to the evidence-retrieval procedure.
Conclusion: Standalone vision–language encoders are a viable retrieval option, but yield somewhat lower downstream answer accuracy in our experiments. We therefore choose shallow VLM retrieval to balance retrieval quality and processing cost. We evaluated CLIP ViT-L/14@336 and SigLIP alongside shallow VLM retrieval, and found that cross-layer token voting gives higher downstream answer accuracy in our setting: 59.48%, compared with 57.47% for SigLIP and 56.61% for CLIP, with diversity selection and temporal expansion disabled (Figure 6). These comparisons motivate our choice of shallow VLM representations for query-relevant evidence selection.
Analysis: A plausible explanation is that retrieval benefits from both richer semantic knowledge and a more contextualized understanding of the question. CLIP and SigLIP encode images and text separately and rank them using global embedding similarity. Our retriever instead uses the VLM's language-model representations to contextualize the query and aggregate token-level relevance signals across shallow layers. This may help identify specific objects, actions, or relations required for answering, beyond broad image–text similarity.
Early-Layer Vision Truncation: This is a promising complementary direction. However, CLIP/SigLIP are trained to align their final image and text representations, so retrieval with intermediate visual features would require checking their alignment with the text encoder and potentially adapting it. Our current comparisons use the full pretrained encoders. ShallowStream retains training-free compatibility with existing VLMs and already achieves 10.29–12.81 ms/frame online processing with the full vision encoder included; combining shallow language-model indexing with an adapted lightweight vision encoder is a useful direction for further efficiency improvements.
Conclusion: The history budget B balances detailed historical coverage and GPU-memory efficiency: it determines how much history remains uncompressed before older units are consolidated into long-term clusters. Our preliminary budget sweep identified B = 64 units as a practical operating point for continuous streaming, and we keep it fixed in all LC-on experiments (Table R16). This corresponds to 128 sampled frames on Qwen3-VL.
Setup & Results: Fixing Qwen3-VL, 1 FPS, LC-on, the standard Gate, end-of-video queries, and a 32-frame prefill batch while varying only B (Table R16), larger B retains more detailed history and raises accuracy modestly from 47.80% to 48.80% at the cost of higher GPU memory, from 18.38 to 19.09 GiB. We therefore choose B = 64 for continuous streaming deployment and have added this setting to Implementation Details in Section 5.1.
Table R16: Effect of the detailed-history budget on GPU memory and accuracy.
Conclusion: There is a misunderstanding here. Section 4.2 performs unit-level selection rather than multi-stage token pruning. Here, 4K denotes a 4 × K = 32 candidate pool, giving diversity selection headroom before narrowing to K = 8 units. We have expanded the description in Section 4.2 to clarify these steps and improve readability.