acceptodds
Under review as a conference paper at ICLR 2027

ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding

Abstract

Streaming video understanding requires models to continuously retain useful visual evidence before future questions are known. Existing approaches primarily manage the growing visual context according to token importance, temporal redundancy, or segment-level relevance, but rarely organize evidence around objects that persist and evolve over time. Thus, in this paper, we introduce ObjectStream, a training-free framework that treats latent objects as memory anchors for streaming video understanding. ObjectStream induces spatially coherent latent objects directly from frozen Video-LLM representations, links them across frames into persistent anchors, and maintains their histories under a bounded memory budget, without requiring external object detectors or segmentation models. Built on these anchors, ObjectStream preserves three complementary forms of evidence: persistent object histories, transient object changes, and recent visual context. This design enables existing Video Large Language Models (Video-LLMs) to reason over object identities, interactions, and state changes while leaving the underlying model unchanged. Extensive experiments on online streaming and offline long-video benchmarks demonstrate both effectiveness and efficiency. In online streaming evaluation, ObjectStream improves Qwen2.5-VL-7B by 10.0 points on OVO-Bench Real-Time Visual Perception, while reducing peak GPU memory and TTFT by approximately 50%. On offline long-video benchmarks, it surpasses the full-token baseline while discarding 82.5% of visual tokens. These results highlight latent objects as a practical and effective organizing principle for compact streaming video memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming video understanding, object centric

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZBgyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents ObjectStream, a training-free visual-memory framework for streaming video understanding. The method identifies salient patches through their dissimilarity to the frame-level feature average, groups spatially connected patches into latent object candidates, and associates these candidates across frames using feature similarity and exponential moving averages.

Critical strengths
  1. The method offers a simple way to construct object-like memory without additional learned models.
  2. Reported improvements over the original backbones are substantial on online perception.
  3. The paper includes useful construction and sensitivity comparisons.
Critical weaknesses
  1. The claimed bounded streaming memory is not established by the described algorithm. Bounding the number of tracks and the final tokens exposed to the language model does not necessarily bound the state maintained during ingestion.
  2. The evidence does not adequately isolate the benefit of persistent object identities. Cross-frame matching uses independent nearest-neighbor feature similarity without a stated one-to-one assignment constraint or spatial/motion consistency requirement.
  3. The efficiency comparisons require a clearer accounting of computation and retained context. Table 5 reports 16.223 s TTFT for Patch Top-K versus 6.697 s for ObjectStream, despite identical peak memory and approximately matched token budgets. It also reports substantially different TPOT values across object-construction variants, ranging from 0.240 s to 1.230 s. Since these variants share the language-model backbone, these differences require an explanation in terms of actual context lengths, caching, execution paths, or measurement boundaries.
Questions for authors
  1. What exactly is stored in each historical token pool during ingestion? Is it incrementally pruned, and how are temporal-event buffers capped and evicted?
  2. How does ObjectStream perform when cross-frame association is disabled or randomized while preserving the same spatial candidates, recent window, and total token budget?
  3. Why is Patch Top-K substantially slower than ObjectStream in Table 5, and why do the object-construction variants have such different TPOT values?
Reviewer_4MEDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ObjectStream, a training-free framework that organizes streaming visual memory around latent object anchors. It groups salient visual tokens into spatially connected regions, associates these regions across frames, and retains representative evidence for each maintained track. Object-conditioned temporal residuals preserve evidence around abrupt changes, while a recent visual window retains detailed observations. Experiments on streaming and offline video benchmarks show improvements over the Qwen2.5-VL backbones, particularly on OVO-Bench real-time perception, together with reduced memory usage and response latency in the reported online evaluation.

Critical strengths
  1. Promising empirical gains with useful supporting analyses. ObjectStream improves Qwen2.5-VL-7B by 10.0 points on OVO-Bench real-time perception and by 6.1 points over FluxMem. Component ablations, object-construction alternatives, and sensitivity analyses provide useful evidence beyond the main results. The reported online memory and latency reductions further support its practical relevance.

  2. A practical training-free memory design. Persistent object evidence, transient changes, and recent observations serve complementary purposes. Constructing object-like regions directly from frozen backbone features avoids additional detectors, segmenters, or model training, making the approach practically appealing for adapting existing video-language models.

Critical weaknesses
  1. The specific benefit of persistent object anchors remains insufficiently established. The paper does not quantitatively evaluate association consistency, fragmentation, or incorrect merging. Moreover, Patch Top-K already achieves 72.8 on OVO-Bench real-time perception, compared with 73.3 for ObjectStream. While ObjectStream also reports a substantial TTFT advantage over Patch Top-K, the small accuracy gap leaves unclear how much of the QA improvement specifically comes from cross-frame object association rather than selective token retention. A budget-matched comparison with spatial grouping without cross-frame association would more directly test the central hypothesis.

  2. The resource guarantees and configuration choices need stronger documentation. The method accumulates historical token pools and subsequently selects representative tokens, but does not clearly specify whether all persistent buffers are bounded online. The offline configuration also substantially increases object-memory capacity and retains 80 recent frames, while the reported latency and memory measurements are limited to OVO-Bench. Although the appendix describes a held-out grid search, the selection of the separate online and offline configurations remains insufficiently detailed. These gaps limit reproducibility and confidence in the claimed efficiency across operating regimes.

Questions for authors
  1. Effectiveness of latent objects. Under matched token budgets, how does ObjectStream compare with spatially grouped regions without cross-frame association? On a representative subset, can you report association failures such as incorrect merging, fragmentation, or identity switches, and relate them to downstream QA performance?

  2. Hyperparameter selection. Please specify the size and evaluation separation of the held-out StreamingBench split, the search ranges, selection criterion, and backbone used. Were the substantially larger offline settings selected using the same procedure, and were they fixed before evaluating the offline benchmarks?

  3. Storage and offline efficiency. Are historical token pools and temporal-event buffers pruned during ingestion? Please clarify their maximum storage requirements and report actual retained-token counts, peak memory, and processing latency on VideoMME-Long, alongside the dense backbone and FluxMem under a common protocol. How do these costs change with input length?

  4. Scope of generalization. Which parts of ObjectStream depend on Qwen2.5-VL-specific visual representations or positional handling? If available, a small evaluation on an architecturally different backbone with unchanged memory rules would help clarify the scope of transferability.

Reviewer_uwG7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ObjectStream proposes a training-free method for enhancing the streaming video understanding performance. The method contains three main components to find object histories, their changes and recent visual context. The evaluation on Qwen2.5-VLs demonstrates the effectiveness of the method.

Critical strengths
  1. The paper is clear and easy to follow. The design of the three components is complementary and reasonable.
  2. The comparison is comprehensive and includes state-of-the-art baselines.
Critical weaknesses
  1. Apart from the three higher-level components, the authors also designed S1-S4 in LOAM. But the ablations of these subcomponents are missing.
  2. The authors only used Qwen2.5-VLs in the evaluation, which does not sufficiently support the " plug-and-play " claim.
  3. I think the authors should rethink the presentation and explanation of table2. Based on the results discussion, it seems the authors only focus on the lower half of the table. And there are also a few interesting observations in table 2: firstly, many 7B-models outperform Qwen2.5-VL-7B with/without ObjectStream, and this even includes Qwen2-VL-7B. Can the authors explain this observation? Secondly, FluxMem's performance is closer to ObjectStream on 7B model, so is it possible that the advantages of ObjectStream are only obvious on smaller Qwen2.5 VLs? That's also why I suggest to include more models with your method.
  4. Some writing and format suggestions: a. Can you change the citation format? The current one makes some content hard to read; b. Line196 has Fig. ??; c. The captions of tables should be on top of the tables.
Questions for authors

My main questions have been asked in the third point of the weaknesses, please check them. And apart from the designed techniques, could you please clarify the novelty of your method?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review and Rebuttal Summary

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs, We sincerely thank all Area Chairs and all reviewers (uwG7, 4MED, and ZBgy) for their time and constructive feedback that helped us improve the paper. We have uploaded the revised manuscript and supplementary materials with all major updates highlighted in blue.

1. Overview of Review Status

  • Score Progression: Our initial ratings were [2,2,3] and currently stand at [3,3,3] (Weak Accept)
  • Reviewer Engagement:
    • Reviewer uwG7 and 4MED confirmed that our additional experiments resolved all concerns and raised their score from 2 to 3.
    • Reviewer ZBgy confirmed that our new results addressed their concern and strengthened the paper, maintaining their recommendation for acceptance.

2. Summary of Key Revisions and Clarifications

Through the rebuttal and manuscript revision, we resolved all raised concerns across three main fronts:

A. New Experiments & Architectural Validation (Added & Modified)

  • LOAM Sub-Component Ablations (S1–S4): Added four single-component ablations isolating steps S1–S4 on OVO-Bench Realtime in Appendix §7.10 (W1, Reviewer uwG7).
  • Cross-Backbone Generalization (Qwen3.5-9B): Evaluated ObjectStream out-of-the-box—without modifying core memory rules—on Qwen3.5-9B across OVO-Bench, StreamingBench, VideoMME-Long, and EgoSchema in Tables 1 and 2 (W2, Reviewer uwG7; Q4, Reviewer 4MED).
  • Controlled Cross-Frame Association Ablations: Added a budget-matched ablation comparing Frame-local, Randomized, and Persistent Objects on OVO-Bench Backward Tracing in Appendix §7.11 (Table 8), directly isolating the benefit of persistent object identities (W1/Q1, Reviewer 4MED; W2/Q2, Reviewer ZBgy).
  • Association Quality Audit & Failure Analysis: Added a 300-link manual association audit (including comparisons against SAM+DINOv2) alongside representative failure-case visualizations in Appendix §8 and Figs. 6–7 (W1/Q1, Reviewer 4MED; W2/Q2, Reviewer ZBgy).
  • Efficiency & Input-Length Scaling Benchmarks: Added Peak GPU Memory, TTFT, and TPOT benchmarks on StreamingBench (Table 6) and input-length scaling evaluations across 128, 256, and 512 frames on VideoMME-Long in Appendix §7.12 (Table 9) (W2/Q3, Reviewer 4MED; W3/Q3, Reviewer ZBgy).
  • Table Restructuring & Formatting Refinements: Restructured Table 2 with explicit paradigm block labels (proprietary, open-source offline, and open-source online), fixed the missing figure reference on Line 196, and placed all table captions above their respective tables per the ICLR template (W3/W4/Q1, Reviewer uwG7).

B. Algorithmic Guarantees & Experimental Protocols (Clarified)

  • Strict Online Memory Bounds: Updated Appendix §7.4 to explicitly specify intermediate ingestion-time buffer contents, incremental pruning/eviction rules, and strict bounded streaming memory guarantees (W2/Q3, Reviewer 4MED; W1/Q1, Reviewer ZBgy).
  • Hyperparameter Selection Protocol: Clarified the online tuning protocol (using a 600-question StreamingBench subset) and confirmed that offline configurations were carried over and locked prior to benchmark evaluation in Appendix §7.4 (W2/Q2, Reviewer 4MED).

3. Core Contributions of ObjectStream

  • Training-Free, Plug-and-Play Object-Centric Streaming Memory: Introduces a training-free object-centric framework that unifies persistent cross-frame object histories (LOAM), Object-Conditioned Temporal Residuals(OCTR), and Recent Visual Grounding Window(RVGW), generalizing seamlessly across distinct VLM architectures (e.g., Qwen2.5-VL and Qwen3.5-9B).
  • Strictly Bounded Memory & Low-Latency Streaming: Guarantees bounded ingestion-time and context-time memory budgets via incremental pruning and eviction, achieving substantial TTFT/TPOT speedups over dense and heuristic token-selection baselines as video length scales.
  • Consistent Gains Across Online and Long-Form Offline Benchmarks: Delivers strong improvements across real-time streaming benchmarks (OVO-Bench, StreamingBench) as well as long-horizon offline video understanding (VideoMME-Long, EgoSchema).

Thank you for your time and assistance.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.