- Rating
- 4
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes OneStreamer, a streaming video LLM built on Qwen3-VL-4B. Using a set of control tokens, it unifies three functions within a single causal generation process: recording evidence, waiting, and responding. The paper makes three main contributions: 1.PHCM: The model proactively generates time-stamped local-detail captions and event-level summaries, which serve as textual memory. At inference, this memory is concatenated with a Recent-N FIFO visual window, so historical visual tokens no longer need to be retained. 2.PSTL: A selective supervision scheme for the state-token loss. All output anchors are kept, while the remaining transition groups are subsampled using a quota defined by the largest number of transitions into an anchor. This alleviates the class imbalance caused by the dominance of silence tokens. 3.OneStreamer-1M: A streaming data synthesis pipeline consisting of a caption branch and a QA branch with response-time calibration. The synthesized data is combined with cleaned open-source data.
Critical strengths
- Strong experiments and valuable data. The empirical results are very strong. The proposed data pipeline and the OneStreamer-1M dataset would be a substantial contribution to the community.
- Solid ablations. The ablations cover both data and method. The control-token ablation in particular offers useful insights into how proactive response is learned.
- Effective PHCM design. PHCM is naturally integrated into training, which avoids the train–inference gap of prior memory methods. Ablations show that it reduces computation and extends the accessible temporal range, while maintaining real-time performance comparable to the recent-window-only baseline.
- Interesting state-transition formulation. Modeling response control as state transitions offers a simple and effective new approach for the streaming community.
Critical weaknesses
- Repetitive caption outputs. Since captions are constructed segment by segment, each segment gets its own independent description. This may lead to repetitive captions when an action spans multiple segments. Unlike humans, the model may not learn to say that an action is still ongoing, or simply stay silent, and may instead describe the same action again in every segment.
- Limited evaluation of long-term memory. The paper does not evaluate on long streams such as hour-long videos, and there is no mechanism to handle the growing caption memory. It is also unclear how accurate the generated captions are, and how errors in memory affect later answers.
- Lack of memory correction. Early segments may sometimes lead to a wrong interpretation, for example when an action or scene is ambiguous at first and only becomes clear later. However, the model has no ability to correct earlier caption records based on later evidence, so such errors remain in memory and may mislead subsequent answers.
Questions for authors
- How are the KV cache and positional encoding handled when evicted frames are replaced by captions, and does training simulate this eviction to avoid a train–inference mismatch?
- Since PHCM is only applied on two benchmarks, does enabling it on the other tasks or in real-world use cause any degradation, for example in proactive response timing or latency?
7 comments
W2: Limited evaluation of long-term memory. The paper does not evaluate on long streams such as hour-long videos, and there is no mechanism to handle the growing caption memory. It is also unclear how accurate the generated captions are, and how errors in memory affect later answers.
We thank the reviewer for raising these concerns about long-term memory evaluation and reliability. We have conducted additional duration-stratified experiments and clarify below how caption memory is constructed, its current capacity limitations, and its utility for subsequent QA.
Scalability of long-video memory: We agree that hour-long video streams are important for evaluating long-term memory. Given the limited availability of standardized hour-long streaming benchmarks, we conducted additional experiments on OVOBench-Backward to examine PHCM across the video durations available in this benchmark. We divided the benchmark into three duration groups and measured memory length, GPU memory consumption, inference latency, and QA accuracy on a single NVIDIA H200 GPU. We compared PHCM with a Full Video baseline that retains the complete observed video history sampled at 4 FPS. The results are shown below:
| Video Length | Avg. Duration | Memory Tokens (PHCM) | GPU Memory (PHCM) | Latency (PHCM) | Full Video Acc. | PHCM Acc. | Δ |
|---|---|---|---|---|---|---|---|
| Short (≤90s) | 37.5s | 295 | 9.076 GiB | 0.118s | 80.5 | 81.1 | +0.6 |
| Medium (90–300s) | 182.4s | 1,319 | 9.374 GiB | 0.155s | 71.6 | 76.1 | +4.5 |
| Long (>300s) | 430.3s | 4,486 | 10.059 GiB | 0.260s | 65.4 | 73.7 | +8.3 |
Across the three duration groups, PHCM's memory length and inference latency increase with video duration, while GPU memory consumption rises from 9.076 GiB to 10.059 GiB. PHCM outperforms the Full Video baseline in all three groups, with the accuracy gap increasing from 0.6 to 8.3 percentage points. These results support the utility of compact textual records for retaining historical information within the evaluated duration range.
Regarding memory capacity, PHCM generates an average of approximately 558.1 memory tokens per minute of video on OVOBench-Backward. The longest video in this benchmark is 19 minutes and 58 seconds and produces 8,920 tokens of textual memory. Extrapolating this average rate to the base model's 256K context window gives a token-budget estimate of approximately 7.6 hours. Our current evaluations do not reach the context limit, and we do not employ an explicit memory eviction policy. A possible extension for longer streams would be to selectively evict earlier local-detail records (</Observe>) while retaining their corresponding event-level summaries (</Summary>) as the context approaches its capacity limit. This could reduce context consumption while retaining coarser descriptions of historical events.
Caption reliability and its impact on subsequent QA: During data construction, we use Gemini and Seed to generate timestamp-grounded captions at multiple temporal granularities. These training annotations are verified against the source videos for factual accuracy and temporal consistency. Captions containing unsupported content, incorrect temporal boundaries, or inconsistencies across annotation granularities are filtered out. At inference, PHCM-generated captions serve as explicit historical context for subsequent QA. Downstream QA therefore provides an end-to-end measure of their utility. In the duration-stratified experiments above, PHCM achieves an accuracy of 73.7 in the Long group, outperforming Full Video by 8.3 percentage points. This supports the usefulness of generated records as historical context over the evaluated video durations.
We will include these duration-stratified experiments and the accompanying discussion in the revised manuscript.
W3: Lack of memory correction. Early segments may sometimes lead to a wrong interpretation, for example when an action or scene is ambiguous at first and only becomes clear later. However, the model has no ability to correct earlier caption records based on later evidence, so such errors remain in memory and may mislead subsequent answers.
We thank the reviewer for pointing out this limitation. PHCM currently uses append-only memory: newly generated timestamped captions are appended to the existing caption history without modifying or deleting previous records. When early observations are ambiguous or incorrectly described, subsequent evidence does not automatically trigger revisions to those records.
Later records can provide additional evidence that may help resolve earlier ambiguities during QA. For example, if an action becomes clearer in later frames, the model can append observations describing that evidence. OneStreamer then answers questions using the accumulated local descriptions, summaries of completed events, and recent visual evidence. Any resolution of conflicting records therefore depends on the model's ability to integrate evidence across time during answer generation. Explicit memory revision remains important directions for future work.
We will clarify the scope of PHCM's append-only memory in the revised manuscript and explicitly discuss the lack of memory revision as limitations of the current approach.
We thank the reviewer for the careful evaluation of our work. We address each concern below.
We thank the reviewer for this comment and clarify how captions are constructed. PHCM does not require a caption for every input segment. The frame- or clip-level captions used to construct
</Observe>targets are associated with annotated temporal intervals, and each target is released only after its supporting interval ends. Caption release times therefore follow the annotated intervals rather than a fixed schedule. At inference, the model conditions its writing decisions on the current visual input and retained textual history, and can generate</Silence>without writing to memory. Access to preceding records allows writing decisions to take earlier observations into account. The model is thus not required to describe each segment independently. In the example reported in Table 19, it generates 26 local-detail records over a 360-second stream, corresponding to approximately one record per 14 seconds on average. This example illustrates that memory writing can be sparse rather than occurring at every update.