acceptodds
Under review as a conference paper at ICLR 2027

StreamingMEM-R1: Learning memory evolution for streaming video understanding

Abstract

Streaming video understanding requires models to interpret incoming content causally, preserve information that may disappear from view, and respond at the appropriate time. Existing approaches typically retain fragmented historical representations and rely on predefined memory-update rules, limiting their ability to track evolving events and coordinate evidence retention with response decisions. To address these challenges, we propose **StreamingMEM-R1**, a framework that jointly learns memory evolution, evidence routing, and response timing. At each streaming step, the model performs structured memory evolution through two complementary modules: stable memory and dynamic memory. The evolved states parameterize a response policy that estimates whether an answer is warranted. For response steps, a routing variable specifies stable, dynamic, or joint memory as the answer context. We construct 100K supervised fine-tuning examples and 50K reinforcement-learning examples to teach these decisions explicitly. Streaming-aware fine-tuning preserves causal visual and semantic context, while memory-oriented reinforcement learning provides verifiable supervision for memory evolution and trajectory-level correctness. Experiments across four benchmarks demonstrate that StreamingMEM-R1 enables effective memory evolution, accurate long-horizon reasoning, and efficient streaming video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming video understanding, Memory evolution, Vision-language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BP1UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose StreamingMEM-R1, a framework that jointly learns memory evolution, evidence routing, and response timing to address the challenges of tracking evolving events and coordinate evidence retention with response decisions. Experiments have demonstrated the effectiveness of the proposed method.

Critical strengths
  1. The paper introduces memory operation and memory routing to logically decouple memory maintenance from question-answering behavior, while enabling joint optimization over complete trajectories.
  2. Extensive experiments demonstrate the superior performance of the proposed method.
Critical weaknesses
  1. StreamingMEM-R1 uses the same video sources and annotations as ThinkStream. However, it remains unclear that whether the performance improvements from the proposed evolution of Stable/Dynamic Memory, higher-quality teacher-generated trajectories, or additional training and verification procedures.
  2. The paper only evaluates performance on downstream tasks without directly assessing the quality of the learned memory, such as memory correctness and memory update error rates.
Questions for authors
  1. Could you further discuss the primary sources of the performance improvements and clarify how much each component contributes to the overall gains?
  2. Could you provide a separate evaluation of the generated memory?
3 comments
AuthorsOpenReview ↗

Response to Reviewer BP1U

Response to Sources of Performance Gains

Sharing video sources and base annotations with ThinkStream does not imply identical supervision or training. StreamingMEM-R1 combines trajectory construction, explicit memory management, and trajectory optimization; we do not attribute all gains to the stable/dynamic memory split alone.

Unlike ThinkStream's compressed reasoning memory, our causal trajectories explicitly encode stable and dynamic memory states, memory operations, answer routing, and response decisions. We filter them for visual grounding, causality, and state consistency. Streaming-aware SFT learns these structured decisions, while memory-oriented RLVR jointly optimizes subsequent behavior using the model's own generated memory.

Ablations using the same training corpus show that the gains cannot be explained solely by changing teacher-generated trajectories:

ConfigurationStreamingBenchChangeOVO-Bench OverallChange
Full StreamingMEM-R180.71-65.83-
Without routing reward76.75-3.9660.62-5.21
Without memory reward77.37-3.3461.73-4.10
Without streaming KV memory70.03-10.6855.63-10.20
SFT only, without GRPO72.62-8.0958.93-6.90

We also added a FIFO baseline with matched recent visual context, semantic history, and historical replay access:

Cache strategyStreamingBenchOVO-Bench OverallLatency/chunkPeak GPU memory
FIFO visual KV + semantic history80.1865.121.91 s31.8 GB
Full StreamingMEM-R180.7165.831.43 s27.6 GB
Full minus FIFO+0.53+0.71-0.48 s-4.2 GB

Additional reward ablations further separate the effects of the training objectives:

Reward configurationStreamingBenchChangeOVO-Bench OverallChange
Full reward80.71-65.83-
Without routing reward76.75-3.9660.62-5.21
Without memory reward77.37-3.3461.73-4.10
Without timing reward78.26-2.4563.94-1.89
Without answer reward73.48-7.2359.17-6.66
SFT only, without GRPO72.62-8.0958.93-6.90

Together, these results support complementary contributions from memory supervision, context maintenance, cache management, and trajectory optimization. They support the full framework.

Response to Direct Memory-Quality Evaluation

We used GPT-6 Astra to evaluate model-generated memory directly:

MetricStreamingBenchOVO-Bench
Stable-memory factual accuracy93.6%91.8%
Dynamic-memory factual accuracy95.1%93.7%
Overall memory factual accuracy94.4%92.8%
Memory-update error rate5.3%6.8%

We define factual accuracy as the fraction of generated atomic facts supported by causally visible video evidence. The memory-update error rate is the fraction of executed update steps containing at least one update error.

Dynamic memory is slightly more accurate, consistent with recent frames directly supporting ongoing actions and states. Stable memory requires compression across chunks, which may lose precise text, attributes, or event details. The lower accuracy on OVO-Bench may reflect its more demanding historical reasoning, including past locations, action order, and insufficient-evidence judgments. These measurements provide direct evidence of memory quality while also identifying remaining update errors.

Thank you for author's response. I have no concerns and decide to keep the positive score.

Dear Reviewer BP1U

We sincerely appreciate your positive feedback. We are pleased that our response has addressed your concerns, and we are grateful for your recognition and support of our work.

Best regards,

Authors

Reviewer_1fRWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper intoduseas Streaminig MEM-R1. A memory streaming model that does long video understanding relying on a structured memory that addresses 2 main concerns from previous methods: 1) explicit semantic evolution and 2) unified trajectory supervision. The backbone of Streaming MEM-R1 is a Qwen-2.5-VL 7B. The method includes three main components: 1 - Memory trajectory construction where a teacher model is used to create ore re-shape existing long-video data into a format approprate for a memory-equiped long video model. 2 - Streaming aware SFT where the model is trained to imitate the teacher's step-by-step trajectories directly from video chunks. 3 - Memory-oriented RLVR, where reinforcement learning is used to address test-train discrepency.

Critical strengths

1 - Explicit modelling of semantic evolution. Memory is rewritten at every chunk, including silent ones, so history is kept as an evolving state rather than an accumulating log. 2 - Interpretable, well-structured memory with clear operations. The stable/dynamic split is intuitive, persistent facts and completed events on one side, ongoing state on the other. The operation set is small and explicit. 3 - Structured trajectory supervision. Every step's decisions (operation, memory content, silence vs. response, route, answer) are supervised, which turns streaming memory into a sequence of discrete, learnable decisions instead of an implicit by-product of reasoning. 4 - Mitigating lossy text memory. This is one og the main issues of text-driven memory modules and the proposed route-triggered historical visual replay lets the model recover visual details.

Critical weaknesses

I think the critical weaknesses here are 2, and mostly regard the way in which the work is compared to previous art. 1 - ThinkStream has also a 7B model while the data reported in these table is agains the 3B models which makes the comparison unfair given the 7B of Streaming MEM-R1. To be specific, on OVO-bench, ThinkStram-7B obtains and average score of 64.9. Streaming MEM-R1 is still better but with less than 1 accuracy points.

2 - Why all experiments are done on Qwen 2.5, a relatively old model. For examples, Qwen 3.5 9B without SFT or RLVR already performs better in streammingBench. Also, Qwen3-VL-8B + MEMO (https://arxiv.org/html/2609.38900), a training-free method, outperforms Streaming MEM-R1 in OVO-Bench (perception only) with 76.0% and in StremingBench real-time with 83%.

Without clarifying this points, while the ideas are interesting, I see no clear benefits of using them. If the authors can adress these concerns I am open to give a positive score.

Questions for authors

Check above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1fRW

Response to ThinkStream-7B Comparison

We agree that comparison at the same model scale is necessary. We include ThinkStream-7B below, alongside our Qwen2.5-VL-7B model, on StreamingBench.

MethodBackboneOPCRCSATPEUTRPRSUACPCTAvg.
ThinkStream-3BQwen2.5-VL-3B83.7470.3176.0382.6976.7378.1976.8570.7374.4345.2175.00
ThinkStream-7BQwen2.5-VL-7B85.2375.8180.4384.1578.0080.1580.6872.8376.2050.3777.50
StreamingMEM-R1Qwen2.5-VL-7B86.7379.1788.0387.2474.3781.2286.2385.3175.1463.6380.71

StreamingMEM-R1 exceeds ThinkStream-7B by 3.21 percentage points on StreamingBench. This demonstrates that our method still holds a significant advantage over StreamingBench. We will incorporate these new comparison results into the main results.

Response to Newer Backbones and MEMO

First, our results on the 7B model have already surpassed those of MEMO.

On OVO-Bench Real-Time, using Qwen2.5-VL-7B:

MethodOCRACRATRSTUFPDOJRAvg.
MEMO75.8061.5078.5053.4069.3063.0066.90
StreamingMEM-R182.8273.2172.9363.9972.4768.7072.35
Difference+7.02+11.71-5.57+10.59+3.17+5.70+5.45

On StreamingBench Real-Time, using Qwen2.5-VL-7B:

MethodOPCRCSATPEUTRPRSUACPCTAvg.
MEMO81.0078.1083.9088.0075.8091.0086.1075.2070.7044.7078.50
StreamingMEM-R186.7379.1788.0387.2474.3781.2286.2385.3175.1463.6380.71
Difference+5.73+1.07+4.13-0.76-1.43-9.78+0.13+10.11+4.44+18.93+2.21

With the same backbone, our method improves the reported averages by 5.45 and 2.21 points, respectively, and performs better on 12 of 16 tasks. The largest gains include action recognition, spatial understanding, and counting. These results are consistent with our design: dynamic memory tracks ongoing actions and state changes, stable memory retains key facts and event history, and routing retrieves relevant evidence across chunks.

Then, We provide comparisons with MEMO using the same backbone and additional experiments with Qwen3-VL-8B.

The gains also extend to Qwen3-VL-8B (1fps). On StreamingBench Real-Time:

MethodBackboneOPCRCSATPEUTRPRSUACPCTAvg.
MEMOQwen3-VL-8B89.7075.8090.9090.3079.6095.6083.3074.4082.4052.1083.70
StreamingMEM-R1Qwen3-VL-8B90.3681.6392.1787.5183.7488.3386.4889.2786.9369.9386.77
Δ vs. MEMO—+0.66+5.83+1.27−2.79+4.14−7.27+3.18+14.87+4.53+17.83+3.07

On OVO-Bench Real-Time:

MethodOCRACRATRSTUFPDOJRAvg.
MEMO83.9074.3078.5067.4076.2075.5076.00
StreamingMEM-R179.4385.4680.8374.7972.3178.1778.50
Difference-4.47+11.16+2.33+7.39-3.89+2.67+2.50

Our Qwen3-VL-8B model exceeds MEMO by 3.07 points on StreamingBench and 2.50 points on OVO-Bench Real-Time. We have added these experiments to the appendix and included MEMO as an important baseline in Related Work.

We also compare against the Qwen3.5-9B base model.To ensure fairness, we used a uniform frame rate of 1 fps. The task columns and final average below refer to OVO-Bench Real-Time.

MethodBackboneStreamingBenchOCRACRATRSTUFPDOJROVO Avg.
Qwen3.5-9BQwen3.5-9B81.983.962.470.746.176.270.668.3
StreamingMEM-R1Qwen3-VL-8B86.7779.4385.4680.8374.7972.3178.1778.50

These results show that our gains extend beyond Qwen2.5-VL: the trained Qwen3-VL-8B model outperforms both MEMO with the same backbone and the Qwen3.5-9B base model on the reported averages. We will incorporate the results from the new foundation model, as well as those from MEMO and Qwen3.5-9B, into our main results.

Clarification of Contribution.

Beyond benchmark accuracy, we introduce a unified framework that jointly learns memory routing, memory updates, chunk-level reasoning, and proactive responses. This coordinated approach to memory management offers a promising solution for long-horizon video understanding. We believe our approach has value and hope that the reviewers will recognize the value of our method and re-evaluate our paper.

Reviewer_1fRWOpenReview ↗

Response to authors

I thank the authors for their answer. It makes the things a bit more clear. I am increasing the score to weak rejection. While the work has improved with the provided examples, I believe these comparisons should have been on the paper already. That is my perspective.

Dear Reviewer 1fRW

Thank you very much for your thoughtful feedback and for reconsidering your evaluation of our work. We sincerely appreciate your recognition that the additional examples and comparisons provided in our response have helped clarify our method.

We fully agree with your suggestion that these comparisons should have been included in the original manuscript. Taking your feedback into account, we have now incorporated the additional comparative experiments and analyses into the revised paper, including comparisons with Qwen3.5-9B and MEMO, making the presentation more comprehensive and the empirical evidence more accessible to readers. All revisions are highlighted in blue.

We sincerely hope that these revisions address your remaining concerns and demonstrate our commitment to improving the quality and clarity of the paper. In light of these improvements, we would greatly appreciate it if you could reconsider your current assessment and potentially further increase your score.

Thank you again for your time and constructive suggestions.

Best regards,

Authors

Reviewer_bPoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces StreamingMEM-R1, a framework for streaming video understanding that jointly learns memory evolution, response timing, and evidence routing. At each video clip, the model maintains a dual semantic memory (static and dynamic) and routing process. To train the base model (Qwen2.5-VL-7B), the authors use a teacher model (Qwen3.5-35B-A3B) to construct 100K SFT and 50K RL trajectory samples, combining streaming-aware SFT and memory-oriented RLVR. Experiments on OVOBench, StreamingBench, and other streaming benchmark show strong accuracy improvements.

Critical strengths
  1. Formulating streaming memory management as a sequential decision process with explicit semantic state updates and answer-time evidence routing is well-motivated and conceptually interesting.

  2. The proposed 7B model achieves consistent state-of-the-art results among open-source streaming models across four benchmarks.

  3. While strong performance, it also achieves substantial inference speedups and memory reductions through its bounded streaming window.

Critical weaknesses
  1. The framework relies heavily on fine-grained trajectory labels (memory operations, routing decisions, and response timing) synthesized by Qwen3.5-35B-A3B purely from text captions. Given that this teacher model is relatively lightweight, relying solely on outcome-based rejection sampling is insufficient to guarantee the fidelity of intermediate steps. Evaluating or filtering a subset of these trajectories using a stronger frontier critic model (e.g., Gemini or GPT models) is necessary to validate the quality of the supervision data.

  2. In the memory routing mechanism, selecting the BOTH option provides unrestricted access to both short-term and long-term memory. Consequently, the model has a natural incentive to default to BOTH as a safe shortcut rather than learning precise routing. The paper lacks an empirical analysis of the routing distribution.

  3. In Figure 4, the "w/o KV cache" variant removes the dual-level KV cache entirely, which serves as an overly weak baseline. To meaningfully demonstrate the efficacy of the proposed caching strategy, it should be compared against a realistic, standard baseline such as First-In-First-Out visual window-based KV caching.

  4. Although the four-part composite reward function in GRPO is presented as a core contribution, the paper lacks an ablation study isolating the impact of each individual reward term. It remains unclear how much each reward component contributes to final performance.

  5. The experimental evaluation in Tables 1 and 3 relies on older baselines and omits the latest frontier proprietary models (such as the most recent Gemini and GPT families).

Questions for authors
  1. What is the actual trigger distribution of STM, LTM, and BOTH in both the constructed training dataset and during test-time inference? How do you prevent the teacher model from taking a shortcut and routing almost everything to BOTH?

  2. How do you ensure the accuracy of the intermediate trajectory labels generated by Qwen3.5-35B-A3B, given that your rejection sampling pipeline only verifies the final outcome?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bPoP (Part 1/2)

We have revised the manuscript in response to the reviewers' comments conducted additional experiments. All revisions are highlighted in blue.

Response to Trajectory Supervision Quality

We need to clarify that our rejection sampling is not based solely on rewards for answer accuracy; rather, we take a variety of factors into account, specifically:

The teacher receives the current video chunk, previous memory, causally available queries, and historical video windows when needed (Appendices A.2 and D.1). Validation checks visual grounding, temporal causality, routing–evidence alignment, memory-state consistency, output format, and the response lifecycle. Failed candidates are regenerated using validation feedback or discarded if they cannot be repaired (Appendices A.3, A.6, and D.1). LTM examples additionally require sufficient historical evidence and insufficient current evidence; BOTH examples require joint support, with neither source sufficient alone (Appendix A.5).

From approximately 800K candidate chunk records, we retain 150K after validation, repair, quality ranking, and distribution balancing for SFT and RLVR (Appendix D.1). Given the massive scale of the data, using models like GPT-Astra or Gemini directly would consume an enormous number of tokens, resulting in prohibitively high costs. After weighing the options, we use the locally deployable Qwen3.5-35B-A3B to keep multimodal generation and iterative validation feasible at this scale.

To assess possible biases of our used teacher model, we independently audited 5,000 randomly sampled accepted records with GPT-6 Astra.

Audit outcomeOriginal filteringGPT-6 Astra audit
Records evaluated5,0005,000
Passed all applicable checks5,0004,856
Overall pass rate100.00%97.12%
Failed at least one check0144
Audit criterionPassedPass rate
Visual grounding4,956/5,00099.12%
Temporal causality4,985/5,00099.70%
Memory-transition consistency4,937/5,00098.74%
Routing compliance4,946/5,00098.92%
Response decisions and timing4,913/5,00098.26%
Answer and output-structure compliance4,983/5,00099.66%
All applicable checks4,856/5,00097.12%

The independent audit supports the quality of intermediate supervision, with 97.12% passing all applicable checks and individual pass rates of 98.26%–99.70%. This indicates that by incorporating certain intermediate validation logic, it is possible to align data-filtering capabilities as closely as possible with GPT-6-Astra.

Response to BOTH Routing and Route Distribution

BOTH is constrained by evidence requirements: historical and current evidence must jointly support the answer, and neither source alone may suffice. The routing reward also checks agreement with the reference route, so always selecting BOTH is penalized on STM and LTM examples.

Appendix Table 5 reports the training distribution. BOTH accounts for 4.78% of SFT answer steps. During the RLVR phase, we intentionally increased this ratio because we need more of this type of data to enable the model to learn how to effectively utilize this pattern of combined memory.

Training stageBOTH answer stepsAll answer stepsBOTH share
SFT48310,1044.78%
RLVR5004,05712.32%
Total98314,1616.94%

We additionally measured routing decisions at response time on StreamingBench:

TaskQuestionsSTMLTMBOTH
OP367337246
CR128723917
CS31719010522
ATP303277206
EU159914820
TR321289266
PR108652815
SU246222186
ACP353321248
CT1931007716
Total2,4951,964409122
Share-78.72%16.39%4.89%

Only 4.89% of responses use BOTH. This distribution provides direct evidence against collapse to BOTH as a default shortcut. This indicates that our model only selects the "both" route when it encounters complex problems that may require the use of composite memory, rather than using it as a convenient shortcut.

Response to the FIFO KV-Cache Baseline

We added a FIFO visual-window KV-cache baseline with the same recent visual window, semantic history, historical replay access, model, and input budget as the full method. This comparison helps distinguish the benefits of retained context from those of cache management.

Cache strategyStreamingBenchOVO-Bench OverallLatency/chunkPeak GPU memory
FIFO visual KV + semantic history80.1865.121.91 s31.8 GB
Full StreamingMEM-R180.7165.831.43 s27.6 GB
Full minus FIFO+0.53+0.71-0.48 s-4.2 GB

The dual-level KV cache improves accuracy by 0.53 and 0.71 points while reducing latency by 0.48 s per chunk and peak GPU memory by 4.2 GB. These results support the efficiency of our visual and semantic cache management under matched context access.

AuthorsOpenReview ↗

Response to Reviewer bPoP (Part 2/2)

Response to Reward-Component Ablations

We added ablations that remove each reward component individually:

Reward configurationStreamingBenchChangeOVO-Bench OverallChange
Full reward80.71-65.83-
Without routing reward76.75-3.9660.62-5.21
Without memory reward77.37-3.3461.73-4.10
Without timing reward78.26-2.4563.94-1.89
Without answer reward73.48-7.2359.17-6.66
SFT only, without GRPO72.62-8.0958.93-6.90

Removing any component reduces performance on both benchmarks. The answer reward has the largest individual effect, consistent with its direct supervision of task correctness. Routing and memory rewards also make substantial contributions, while the timing reward yields smaller but consistent accuracy gains. These ablations complement the reward-optimization curves in Figure 3 and support the contribution of each component.

Response to Frontier Proprietary Baselines

We have added a base model trained as Qwen3-VL-8B, which demonstrates superior performance compared to the Qwen2.5-VL-7B model. Consequently, we are using this 8B-parameter model for comparison against GPT-6-Astra. To ensure fairness, we used a uniform frame rate of 1 fps.

On StreamingBench:

MethodBackboneOPCRCSATPEUTRPRSUACPCTAvg.
GPT-6 Astra-94.3686.7893.4094.1786.6393.8384.2589.5391.7564.8389.70
StreamingMEM-R1Qwen3-VL-8B90.3681.6392.1787.5183.7488.3386.4889.2786.9369.9386.77

On OVO-Bench Real-Time:

MethodBackboneOCRACRATRSTUFPDOJRAvg.
GPT-6-Astra-90.888.491.683.777.392.187.3
StreamingMEM-R1Qwen3-VL-8B79.4385.4680.8374.7972.3178.1778.50

GPT-6-Astra achieves higher averages in these comparisons. Our 8B model trails it by 2.93 points on StreamingBench and 8.80 points on OVO-Bench Real-Time. We acknowledge this remaining gap. We look forward to continued advancements in the field of streaming video understanding and believe that in the near future, more advanced methods will emerge that can align with or even surpass the performance of gpt-astra using smaller models. Our results demonstrate gains from the proposed memory and training framework within the evaluated open-model setting.

Clarification of Contribution.

We pioneer a memory routing and update mechanism for streaming video understanding, and introduce a chunk-level memory update and proactive response mechanism. By integrating advanced KV cache management techniques, we believe this approach holds significant value, particularly for long-duration video understanding tasks. We hope that reviewers will affirm its value and re-evaluate our work.

AuthorsOpenReview ↗

Awaiting Your Thoughts on Our Rebuttal

Dear Reviewer bPoP

Thank you again for your initial review. As the discussion phase is nearing its end, we kindly wonder if you had a chance to consider our rebuttal.

We would greatly appreciate any further feedback you might have.

Best regards,

Authors

Comments on the paper

1 comment
Area_Chair_Lm4tOpenReview ↗

Discussion Reminder

Dear reviewers,

the authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you. AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.