acceptodds
Under review as a conference paper at ICLR 2027

SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding

Abstract

Streaming video understanding requires answering questions from the observed video, with evidence drawn from recent observations or earlier events. Historical context can improve recall but may interfere with judgments based on recent frames. Its usefulness depends on both when it is introduced and which clips are selected, while relevance alone does not guarantee sufficient evidence for an answer. We propose SPARE, a framework that combines adaptive historical retrieval with visual encoder sharing. A lightweight gate predicts whether recent observations provide sufficient evidence to answer the question. When additional evidence is predicted to be necessary, a multimodal embedding model retrieves relevant historical clips to supplement the current context. LoRA instruction tuning trains the understanding model to use recent observations and retrieved history jointly and abstain when the evidence remains insufficient. To reduce redundant visual parameters, the understanding and retrieval branches share a visual encoder while retaining separate language backbones. Four linear adapters align the shared features with the retrieval model through distillation at the visual token and final retrieval embedding levels. With visual-only inputs, SPARE achieves 77.78% on the evaluated Real-time and Backward task groups of OVO-Bench and 73.31% on StreamingBench. Compared with Qwen3-VL-8B-Instruct evaluated at 2 fps, SPARE improves accuracy by 20.68 percentage points on OVO-Bench and 14.15 percentage points on StreamingBench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming video understanding, Adaptive Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xUiLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPARE combines evidence-sufficiency gating, historical retrieval, and LoRA tuning to answer streaming-video questions or abstain when evidence is insufficient. The understanding and retrieval branches share a visual encoder through four distilled linear adapters while retaining separate language backbones. Experiments on OVO-Bench and StreamingBench show improved accuracy over the evaluated base model and recent-window baseline, with a compact answering context.

Critical strengths
  1. The selectively retrieving historical evidence addresses the tension between recalling past events and preserving current-scene perception, with potential value for real-time interaction.

  2. The experiments are fairly comprehensive and show promising accuracy–context tradeoffs, with gated retrieval reducing history use and SPARE improving over SimpleStream using only 0.78K additional peak-context tokens on average.

Critical weaknesses
  1. The discussion largely reports numerical changes without explaining why LoRA tuning and gate adaptation substantially improve OVO-Bench, particularly hallucination detection, but slightly reduce StreamingBench performance.
Questions for authors
  1. Among questions that trigger retrieval because current evidence is insufficient, how often does history enable a correct answer, and how often does the model still abstain?

  2. Beyond matching the four visual feature interfaces, what justifies the four independent adapters, and what parameter and memory savings does encoder sharing achieve?

  3. Can you break down latency for queries with and without retrieval and quantify the cost of continuously constructing historical memory?

I would consider raising my score if the authors satisfactorily explain the component effects and practical efficiency tradeoffs.

Reviewer_87SyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPARE, a streaming video understanding framework characterized by evidence sufficiency prediction and question-conditioned history retrieval. A lightweight head predicts whether recent observations provide sufficient evidence to answer the given question. If the evidence is insufficient, the framework expands the context by retrieving question-relevant historical observations through multimodal similarity search. The understanding model is fine-tuned using LoRA to interpret the context augmented with the retrieved evidence. The visual encoder is shared between understanding and retrieval branches through four linear adapters.

Critical strengths
  • The paper experimentally investigates how the quality of retrieved clips affects streaming video understanding. Based on these observations, it focuses on two important questions: (i) whether historical evidence should be retrieved and, (ii) if so, which historical observations should be retrieved. This provides a well-motivated problem formulation.
  • SPARE achieves approximately 10% higher accuracy than the baselines on OVO-Bench and StreamingBench, demonstrating the effectiveness and potential impact of the proposed framework.
Critical weaknesses
  • The motivation for the visual encoder sharing between understanding and retrieval branches (contribution 2) is not sufficiently explained. If the primary goal is to improve efficiency in terms of the parameter size or processing speed, a quantitative comparison between the native ViT and shared ViT configurations would be valuable. Specifically, comparing their parameter sizes, processing times, and benchmark accuracy would help clarify the trade-off between accuracy and efficiency.
  • The introduction of LoRA instruction tuning leads to a particularly large relative improvement of approximately 38% on the HLD of OVO-Bench (Table 1). However, the underlying reason for this improvement is not sufficiently discussed. A more detailed analysis of the mechanism behind this substantial gain would help clarify the contribution of the LoRA tuning.
Questions for authors

I would appreciate clarification on the points raised in the Critical Weaknesses section, particularly regarding the motivation and accuracy-efficiency trade-off of visual encoder sharing, and LoRA's contribution to HLD.

Reviewer_uxoeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes a two step approach for answering questions in streaming setup. First, it tries to answers the question using the features of the last frames. A retrieval gate aims to predict the probability if the questions cannot be answered. If it is above a threshold, previous clips that are similar to the current clip are retrieved and the the features of the retrieved clips are added as visual features to answer the question.

Critical strengths

The basic idea is very simple. Instead of retrieving always video clips from the past, the approach has a trigger mechanism for retrieval, such that it is only used when it is needed. This has also the positive effect that the VLM is not confused by retrieved video clips in case of questions that can be answered by the current frames (real-time), whereas the VLM benefits from the retrieved video clips for questions referring to older video clips.

Critical weaknesses
  1. The contribution is limited. Beyond the simple threshold-based triggering mechanism, the approach does not provide anything new. The introduction also needs a major revision. In particular, l.52-l.79 discusses an ablation study in the introduction instead in the experimental section.

  2. Despite the simplicity of the basic idea, the method itself is not well described. Stage 2 in Fig. 2 combines both iterations, which is confusing at the beginning but it gets clearer later on. Some terms are not well explained like "clip-to-embedding path" or "visual tower". The "text path", which is not shown in Fig. 2, is also unclear. The top K clips are based on cosine similarity of the text embedding of the question q_t and the undefined "clip-to-embedding path". It basically assumes that Qwen3-VL-Embedding-2B directly provides the clip for answering the questions. The shared vision encoder is also not well explained and Fig. 3 does not add much. As before, the terminology like "DeepStack" is not well explained. The paper does not motivate the use of four adapters instead of one.

  3. The paper makes strong assumptions:

3a The overall setup seems to assume that a question can be answered by single clips either the current clip or an older clip or the combination of the current and an older clip. The model can thus not answer more complex questions that require reasoning beyond these two clips. Table 3 indeed shows that 1 retrieved clip performs best.

3b The model is trained by annotating for each question if the current clip provides the answer or not. It seems that these labels are automatically obtained by a pre-processing step using Qwen3.8-27B described in the Appendix A.1.2, but this description is not very clear. Nevertheless, the training procedure is unnecessary complex.

3c The results in Table 2 and 7 in the Appendix show that the claims and motivation in the introduction are not supported by evidence. Gating does not perform better than retrieving always one additional clip. The claim "The experimental findings motivate considering both whether additional history is needed and which clips can usefully supplement recent observations." is contradicted by the results in Table 7. Fig. 1 does not support the claims made in the introduction.

4 There are very few ablation studies. Many design choices are not evaluated, for example,

  • The need of four adapters
  • The loss functions and their weights w_j

Summary: The paper lacks a substantial contribution, the assumptions are strong, claims and motivation are not supported by evidence, and the presentation needs a major revision.

Questions for authors

Please try to clarify the issues 2 and 3b in critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.