acceptodds
Under review as a conference paper at ICLR 2027

StreamWeave: Curating Multi-Timescale Visual Evidence for Streaming Video Understanding

Abstract

Multimodal large language models have advanced offline video understanding, but streaming video systems must answer questions using only observations available when each question arrives. As the observed prefix grows beyond the model's finite visual context, different questions make different observations relevant. A fixed frame selection cannot reliably preserve all such evidence. Even when past frames are retained, event descriptions written before a question arrives may not reveal which frames contain the visual evidence it seeks. We introduce StreamWeave, a training-free agent that curates multi-timescale visual evidence for each question. As video arrives, it builds a searchable visual history linked to original frames. At query time, a recent glance and a broader look-back provide the first-call evidence; the MLLM answers or requests additional source frames through history retrieval. Across three streaming video understanding benchmarks and five frozen MLLMs ranging from an 8B dense model to a 397B mixture-of-experts model, StreamWeave achieves competitive results against published streaming methods. Extensive ablation studies show the effectiveness of its components, temporal layout, and retrieval choices.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Streaming Video Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Nm85OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StreamWeave, a training-free framework for streaming video question answering using only observations available when each question arrives. It combines a recent glance, a broader look-back, and on-demand historical retrieval within a bounded visual context. Its main contributions are a coverage-aware visual history with searchable visual keys and source-frame links, visual-encoder-based filtering of redundant recent frames, and model-directed retrieval of original historical frames without relying on pre-generated event descriptions for indexing. A frozen multimodal large language model either answers from the initial context or requests additional evidence for subsequent reasoning. The authors evaluate five frozen models on OVO-Bench, StreamingBench Real-time, and OVBench, using at most 44 images per model call. Compared with model-matched 44-frame uniform-sampling baselines, overall scores improve for all five models on the first two benchmarks and four on OVBench. Validation includes component ablations, matched-frame temporal-layout comparisons, and historical-frame selection experiments with fixed tool decisions and image counts, alongside an analysis of the accuracy–resource trade-off between adaptive and unconditional retrieval.

Critical strengths
  1. Consistent improvements across models and benchmarks. StreamWeave improves overall accuracy over model-matched uniform-sampling baselines across most settings, including all five models on OVO-Bench and StreamingBench Real-time. This supports the practical value of the framework.
  2. Useful component-level analysis. The paper includes ablations on temporal coverage, frame selection, and historical retrieval. Some comparisons control for frame counts and model calls, providing evidence that the improvements are not solely due to additional visual inputs.
Critical weaknesses
  1. Limited novelty over existing methods. The overall framework overlaps with prior streaming-video memory systems, including OASIS (Liang et al., 2026; https://arxiv.org/abs/2604.17052), EventMemAgent (Wen et al., 2026; https://arxiv.org/abs/2602.15329), and Visual Agentic Memory (Li et al., 2026; https://arxiv.org/abs/2605.16481). StreamWeave introduces visual-key retrieval, coverage-aware record updates, and a multi-timescale context. However, while the paper provides useful ablations, it lacks direct comparisons with the corresponding mechanisms in closely related methods under comparable settings. The incremental benefits of these design choices therefore remain insufficiently established.
  2. Insufficient control of inference cost. While the paper includes comparisons with matched model calls and image counts, these are limited to specific components. The full framework is not evaluated against the main baselines under matched total inference budgets, making it difficult to separate the benefits of evidence selection from those of additional inference.
Questions for authors
  1. Technical contribution: The paper describes visual-key indexing and coverage aggregation. How do these differ from the corresponding mechanisms in OASIS and VAM, and what evidence supports their added value beyond the within-framework ablations?
  2. Inference cost: Beyond the controlled comparison in Table 7, could you report the main configuration’s per-question token usage and latency, with shared indexing costs listed separately? Could a budget-matched comparison with the 44-frame baseline clarify how much of the gain comes from evidence selection rather than extra reasoning?
Reviewer_UKr4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents StreamWeave, a training-free framework for streaming video understanding that aims to preserve relevant visual evidence under limited context budgets. The method combines three main components: (1) a coverage-aware visual memory that stores representative frames and allows retrieving the original visual evidence, (2) a multi-timescale sampling strategy combining a dense recent window with a sparser look-back, and (3) query-conditioned historical retrieval, where the MLLM can request additional frames when needed. The authors evaluate the method on three streaming video benchmarks using five frozen Qwen-family MLLMs, showing improvements over uniform sampling across all models on OVO-Bench and StreamingBench Real-time, and four out of five models on OVBench. The paper also includes several ablations investigating the contributions of the individual components.

Critical strengths

The method shows positive results across several benchmarks and model sizes, improving all five models on OVO-Bench and StreamingBench Real-time, and four out of five on OVBench. I think this is a strong point, especially since the method is training-free and uses the same configuration across all experiments. This suggests that the approach could be useful in practice without requiring additional model training which is quite remarkable.

Critical weaknesses

Most of the proposed components seem relatively standard, and I'm not fully convinced that the online memory is actually necessary to obtain the reported improvements. In particular, the ablations show that removing the look-back decreases accuracy by 3.60 points, while removing historical retrieval only decreases it by 1.01 points. While this doesn't fully isolate the contribution of each component, it raises the question of whether a simpler frame sampling approach could achieve similar results. I think this is important since the online memory is one of the main contributions of the paper.

Also, the main model-matched baseline relies on uniform frame sampling, while several more sophisticated training-free methods already exist, such as BOLT, AKS, and HiMu. These approaches could potentially be adapted to the observed video prefix without using future frames. Without these comparisons, it's difficult to understand whether StreamWeave offers a meaningful advantage over existing frame selection methods. This is particularly important because the paper also lacks a complete accuracy/latency analysis that could otherwise justify the added complexity of maintaining an online memory.

Moreover, since the paper focuses on streaming video, I think latency and computational costs are particularly important. The authors report the number of model calls and processed images, but I would have liked to see actual inference latency, memory usage, and the cost of maintaining the visual history. In particular, retrieval requires additional MLLM calls, which might be quite expensive. Without a proper accuracy/latency comparison against simpler alternatives, it's difficult to understand whether the benefits justify the additional complexity and computational overhead. I think this is a significant limitation, especially considering that efficiency is one of the main motivations for developing a streaming-specific approach.

References

Liu et al. (2025). BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding. CVPR 2025.

Tang et al. (2025). Adaptive Keyframe Sampling for Long Video Understanding. CVPR 2025.

Ben-Ami et al. (2026). HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering. arXiv:2603.18558

Questions for authors
  1. Comparison with traditional VideoQA methods. How would StreamWeave compare against stronger training-free frame selection methods for VideoQA, such as BOLT or AKS, when applied to the same observed video prefix and using the same MLLM and similar computational budgets? I think this comparison is important to understand whether the improvements actually come from the proposed online memory mechanism or could be achieved with better conventional frame selection. A competitive result against these methods would help clarify the contribution of the paper.

  2. Computational advantage of online memory. What is the actual advantage, in terms of accuracy and latency, of maintaining the proposed online memory compared to selecting relevant frames from the observed video only when a question arrives? The paper already reports model calls and image counts, but could the authors provide end-to-end latency and computational costs, including the overhead of maintaining and querying the visual history? I think this is particularly important to understand whether the additional complexity is justified for streaming applications.

Reviewer_ZhLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses streaming video understanding, focusing on which temporal information should be leveraged for input to MLLM under a limited input budget. It tackle the problem, where using only recent frames can miss passed events/details, and uniform sampling of the entire video may provide insufficient temporal density for capturing details. To tackle these, the proposed StreamWeave adaptively retrieves historical records via visual embeddings rather than text summaries, and revisit the corresponding visual values. The paper demonstrates its effectiveness across multiple benchmarks/baseline models.

Critical strengths
  • The paper addresses challenges specific to video streaming task, which is reasonable.
  • The proposed approach operates without training scheme, enabling efficient employment.
  • The experiments cover multiple benchmarks and the comparisons are proper.
Critical weaknesses
  • Overall methodology does not fundamentally differ from the OASIS in related work, raising issues of originality.
  • The performance improvements from StreamWeave are mostly marginal.
  • In some cases like Counting, the performance rather tends to decline.
Questions for authors
  • What is the fundamental conceptual/methodological differences between OASIS and the proposed StreamWeave?
  • How the method handles visually similar frames? For example, when similar frames contain different but important visual cues, the proposed visual-key-based approach may fail to capture those since they maintain similar features.
  • The ablation study primarily examines performance degradation when individual components are removed. Does performance improve consistently as the components are added incrementally?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the Program Chairs, Area Chairs, and all reviewers for their time, effort, and constructive feedback. We are encouraged by the reviewers’ recognition of our paper’s strengths and their positive feedback on our targeted additional experiments. Our goal in this rebuttal revision is to strengthen the paper and improve confidence in its results.

Strengths recognized by the reviewers

  • Practical, training-free streaming-video understanding (ZhLY, UKr4). StreamWeave addresses streaming-specific evidence needs without additional training, using one shared configuration across the evaluated models and benchmarks.
  • Consistent improvements across model sizes and benchmarks (UKr4, Nm85). StreamWeave improves all five models on OVO-Bench and StreamingBench Real-time, and four of five on OVBench.
  • Informative component analyses (Nm85). Comparisons controlling frame counts and model calls support the contribution of effective evidence organization beyond simply supplying additional visual inputs.

Questions raised and our responses

  • Methodological novelty and component contributions (ZhLY, UKr4, Nm85). We explained how source-linked visual records and multi-timescale context organization address different evidence needs, supported by direct OASIS/VAM component replacements, cumulative ablations, and task-level analyses. Coverage aggregation reduces retrieved history frames by 38.26% while maintaining accuracy.
  • Stronger baselines and the accuracy–latency trade-off (UKr4). We added same-prefix comparisons with BOLT, AKS, and HiMu, evaluated uniform sampling with 12/20/32/44 frames, and measured HiMu latency. StreamWeave gains 5.32 percentage points over Uniform 44 at approximately 0.70 seconds of additional mean query latency.
  • Whether additional inference explains the gains (Nm85, UKr4). StreamWeave retains a 4.53-point advantage in a full-framework comparison matching calls and image counts. We also report token usage and query latency, with reusable stream-processing and history-maintenance costs listed separately.
  • Similar-frame retention and Counting performance (ZhLY). We clarified the distinction between appearance-based retention and visual-key retrieval, and provided Counting diagnostics examining the roles and trade-offs of self-curation, aggregation, and historical retrieval.
  • Consistency of the Uniform 44 results (Nm85). We traced the scores in Table 1 and Tables 15–16 to their respective sampling and prompting protocols, clarified their experimental purposes, and identified the baseline used in each comparison.

To this end, our final revision will include the following updates:

  • Clearer explanations of the methodological distinction, frame-retention and retrieval mechanisms, and complementary component roles.
  • Additional component comparisons, cumulative and task-level ablations, Counting diagnostics, inference-budget controls, and accuracy–latency analyses in the appendix.
  • Explicit documentation of the Uniform 44 protocols in Table 1 and Tables 15–16, with query costs and reusable stream-processing costs reported separately.

We hope these clarifications and additional results address the concerns raised and strengthen confidence in StreamWeave’s methodological contribution and practical value. We sincerely appreciate the reviewers’ constructive engagement and welcome further questions and suggestions.

Thank you, the authors of submission #3337

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.