acceptodds
Under review as a conference paper at ICLR 2027

Learning Memory Update Policies for Streaming Video Understanding through Reinforcement Learning

16 views

Abstract

Streaming video understanding requires compact memory that preserves useful information as new observations arrive. Existing training-free methods often compress visual memory using heuristic criteria, while conventional KV-cache pruning has limited applicability to the linear-attention layers of hybrid backbones. We propose MURL(Memory Update Reinforcement Learning), a framework that learns visual-token retention policies for recursive memory updates through reinforcement learning. With the vision-language backbone frozen, a lightweight selector scores tokens from retained history and newly arrived observations. A history gate combines current contextual scores with stored importance estimates to guide repeated compression. We train the policy using video-level caption feedback from a frozen backbone. Experiments with Qwen3.5-4B and Qwen3.5-9B on StreamingBench, OVO-Bench and Video-MME under a streaming protocol demonstrate the effectiveness of learned memory updates for streaming video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Memory, MLLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RqCbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MURL, a reinforcement learning-based framework for visual memory management in streaming video understanding. It introduces a lightweight token selector and a history gate to preserve informative visual tokens across repeated memory compression steps. The policy is trained using caption-based rewards from a frozen vision-language backbone. Experiments with Qwen3.5-4B and Qwen3.5-9B on three video understanding benchmarks demonstrate consistent, though generally modest, improvements over adapted memory compression baselines.

Critical strengths

S1. Practical and relevant problem formulation. The paper addresses the important problem of maintaining bounded visual memory under continuous video input. Optimizing token retention across successive, irreversible memory updates is more relevant to streaming applications than one-shot video compression.

S2. Consistent experimental results and reasonable efficiency. The method is evaluated on two backbone sizes and three benchmarks, with additional ablations and efficiency measurements. MURL consistently outperforms the adapted memory baselines in aggregate performance, and Table 7 reports lower memory-maintenance latency than FluxMem and QueryStream.

Critical weaknesses

W1. Limited performance gains and insufficiently controlled baseline comparisons. Although MURL achieves the best aggregate scores within the adapted memory comparison groups, the improvements are generally marginal. For Qwen3.5-9B, the gains over the strongest competing memory baselines are only 0.56, 0.16, and 0.79 percentage points on Video-MME, StreamingBench, and OVO-Bench, respectively. No statistical variance or confidence intervals are reported. More importantly, Appendix A.4 reveals substantial modifications to the competing methods. FluxMem does not follow the same irreversible recursive compression procedure, QueryStream's original thresholding is replaced with Top-K selection, and StreamForest's full hierarchical event memory is not implemented. Matching token counts at query time does not fully establish comparable online memory constraints. Consequently, the results do not conclusively demonstrate superiority over the original methods.

W2. Insufficient isolation and validation of the core methodological contributions. The proposed reinforcement learning-based token selection builds on existing ideas, while the independent benefits of recursive compression training and the history gate are not convincingly established. In Table 3, the single-compression baseline also disables the history gate, introducing a confounding factor. Comparing the no-gate results in Tables 2 and 3 shows that repeated compression training does not consistently improve performance across benchmarks: StreamingBench decreases from 81.68 to 81.32, while OVO-Bench improves only from 58.74 to 58.80.

Furthermore, training simulates only 1–4 compression events, without directly analyzing performance degradation over substantially longer sequences of memory updates. This weakens the evidence supporting the claimed long-term recursive memory advantages.

Questions for authors

Q1. Can the authors clarify how the adaptations of FluxMem, QueryStream, and StreamForest affect their original performance? Can comparisons under consistent online memory constraints be provided? Q2. Can the authors provide a controlled ablation that independently evaluates repeated-compression training and the history gate? How do they explain the inconsistent gains of repeated compression without the gate across benchmarks? Q3. How does performance vary with the number of memory compression steps, particularly for questions requiring information from early video segments? Q4. Given the relatively small performance improvements, can the authors provide statistical uncertainty estimates and end-to-end inference efficiency measurements?

3 comments
AuthorsOpenReview ↗

Response to Reviewer RqCb

We thank the reviewer for recognizing the strengths of our work and for their constructive feedback. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

W1&Q1: Impact of Baseline Adaptations on Performance

We thank the reviewer for raising this concern. Table 1 compares all methods using the same Qwen3.5 backbone, sampled frames, evaluation examples, QA prompts, and decoding settings. We match the visual token count exactly at each query while preserving each baseline’s selection or compression strategy.

QueryStream retains its question-aware importance scores. Top-K selects the highest-scoring tokens under the shared budget, preserving the ranking used by its original threshold rule. The evaluation protocol fixes the answer time, so RTAR’s response scheduling is unnecessary. By “FluxMem does not follow the same irreversible recursive compression procedure,” we mean that, when adapting FluxMem, we followed its original compression strategy more closely instead of applying the repeated, irreversible compression used in MURL. FluxMem retains broader access to past information.

Since it was difficult to preserve StreamForest’s original hierarchical memory updates while matching the visual token budget exactly for every example, we omitted this mechanism from the original comparison. We also evaluated an adaptation that restores hierarchical event memory.

MethodShort (%)Medium (%)Long (%)Overall (%)
StreamForest with hierarchical memory80.4463.6751.7865.29
Fixed-capacity ToMe without hierarchical memory81.6765.2254.5667.15
MURL (ours)81.6771.4460.7871.30

Table 1 thus provides a fair comparison of compression strategies under a shared backbone and query input budget, while preserving QueryStream’s access to the question and FluxMem’s access to the full history.

W2&Q2: Insufficient isolation and validation of the core methodological contributions. We thank the reviewer for pointing this out. We agree that reporting the ablations in two separate tables made the relationship between repeated-compression training and the history gate difficult to see. With the gate disabled, switching from single- to repeated-compression training yielded inconsistent gains across benchmarks. StreamingBench accuracy fell from 81.68% to 81.32%, while OVO-Bench accuracy rose only slightly, from 58.74% to 58.80%. The best results came when we paired repeated-compression training with the history gate. The gate lets each compression step use information about which tokens were important in earlier rounds, and repeated training teaches the model to use that information across successive updates. In the revision, we will present the three settings side by side so readers can see both the effect of switching to repeated training with the gate disabled and the further improvement from enabling the gate.

AuthorsOpenReview ↗

Response to Reviewer RqCb (Part 2)

Q3:How does performance vary with the number of memory compression steps, particularly for questions requiring information from early video segments?

We thank the reviewer for the question. To examine how accuracy varies with the number of compression rounds, we grouped Video-MME questions by the number of rounds MURL performed on each video and compared MURL with the 128-frame baseline on the same questions.

MURL compression roundsQuestionsBase-128f (%)MURL (%)
089783.2881.72
1–21275.0083.33
3–536373.8372.45
6–1043568.9770.34
11–209364.5270.97
>2090060.1160.78
Overall2,70071.3071.30

The accuracy gap showed no consistent trend as the number of rounds increased. Even after more than 20 rounds, MURL scored 60.78% versus 60.11% for Base-128f across 900 questions. The baseline’s advantage in the zero-round group likely reflects its denser sampling of these short videos. These results suggest that combining the history gate with training that simulates 1–4 compression rounds allows MURL to generalize to more compression rounds at inference time.

Current datasets do not provide exact evidence timestamps, preventing direct analysis of how evidence-to-query delay affects performance. We therefore randomly selected 700 OVO-Bench samples with at least 300 seconds of video remaining after the original query time. Since the evidence for these questions appears before the original query, postponing the query by 75, 150, 225, or 300 seconds increases the evidence-to-query delay by the corresponding amount. During this interval, MURL continued processing the video and updating its memory. We then measured accuracy at each delay to assess its impact on performance.

Added query delayMURL (%)
0 s57.14
75 s57.14
150 s57.43
225 s58.00
300 s57.43

Accuracy remained between 57.14% and 58.00% across all delays. This suggests that MURL retains earlier evidence through repeated memory updates.

Q4: Given the relatively small performance improvements, can the authors provide statistical uncertainty estimates and end-to-end inference efficiency measurements?

We thank the reviewer for this suggestion. The MURL accuracies reported in Table 1 of the paper are the means of three independent training runs with different random seeds. All runs used the same training configuration and checkpoint-selection rule, and all evaluations used greedy decoding. Below, we report each run’s accuracy alongside the mean and sample standard deviation to show how performance varied across training seeds on each benchmark.

Training runVideo-MME (%)StreamingBench (%)OVO-Bench (%)
Run 171.1981.9060.03
Run 271.4181.8260.07
Run 371.3081.9259.99
Mean ± SD71.30 ± 0.1181.88 ± 0.0560.03 ± 0.04

We also measured inference efficiency on 200 Video-MME videos using Qwen3.5-9B in BF16 on one NVIDIA H200 GPU, with one warm-up and three timed repetitions per video. All methods compressed memory from 11,520 to 5,760 visual tokens at each update and averaged 21.08 updates per video.

MethodVisual encoding (s)Memory updates (s)Prefill (s)Query TTFT (s)Answer generation (s)End-to-end replay (s)Throughput (frames/s)
MURL11.69541.1530.5162.8760.835126.22425.94
FluxMem (adapted)11.57080.4470.5902.7740.882164.78515.21
QueryStream (adapted)11.26169.0620.4742.4990.735152.81717.40

MURL had the lowest cumulative memory-update time (41.153 s), the shortest end-to-end replay time (126.224 s), and the highest streaming throughput (25.94 frames/s). TTFT runs from query-input assembly to the first generated token, with memory already prepared. Answer generation includes prefill, and end-to-end replay covers video decoding through answer generation. Throughput covers visual encoding, position construction, and memory maintenance. We will include both tables and these timing definitions in the revision.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal

Dear Reviewer RqCb,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results to address your concerns, and incorporated the key updates into the revised manuscript. With the discussion period drawing to a close, we would be grateful if you could revisit our rebuttal when convenient and let us know whether any points need further clarification. We would welcome the opportunity to address any remaining concerns.

We sincerely appreciate your time and consideration.

Best regards,

The Authors

Reviewer_ke2HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MURL, a reinforcement learning framework for query-independent visual memory maintenance in streaming video understanding. With the vision-language backbone frozen, MURL uses a lightweight selector and history gate to retain tokens across recursive compression, and trains the policy through repeated rollouts using full-video caption loss as feedback. Experiments with Qwen3.5-4B and Qwen3.5-9B on three benchmarks demonstrate improvements over adapted memory baselines under matched token budgets.

Critical strengths
  1. The paper is clearly written and well organized. The overview figures effectively explain the streaming update process, selector architecture, and policy training procedure.

  2. The history-aware selector is intuitive and lightweight. Combining current contextual scores with historical importance appropriately addresses the changing context caused by recursive compression.

  3. MURL consistently improves over the adapted memory baselines across two backbone scales and three benchmarks under matched token budgets. Tables 2–4 provide useful evidence supporting history-aware selection and future-aware credit assignment in the complete system.

Critical weaknesses
  1. The query-independent setting is reasonable, but a single full-video caption does not represent the distribution of possible future questions. MURL therefore learns average caption saliency rather than directly optimizing the retention of subtle query-relevant evidence, particularly details absent from the caption. Although Table 1 shows aggregate transfer to QA, Table 4 only compares credit assignment under the same caption reward; it would be better to evaluate frame/segment-level captions, QA-based or mixed supervision, and results by question type.

  2. The inference-efficiency evaluation is incomplete. Table 1 reports only accuracy, while Table 7 measures memory-maintenance time for one update cycle without hardware details or end-to-end latency. It would be better to supplement the controlled comparisons in Table 1 with Table 7’s overhead metrics and report streaming throughput and query latency, including visual encoding, repeated updates, VLM prefill, and answer generation.

  3. The controlled comparisons in Table 1 rely on substantially adapted baselines: Appendix A.4 states that FluxMem does not reproduce irreversible updates, QueryStream is question-aware, and PEMF omits its hierarchical mechanism. Matching retained token counts therefore does not ensure equivalent information access or memory constraints, which is important because the 9B gains over the strongest adapted baselines are only 0.56, 0.16, and 0.79 points on the three benchmarks. Simple causal fixed-budget baselines or more faithful implementations would make the conclusions more convincing.

  4. Table 3 evaluates the joint change from single-compression training without the history gate to the complete repeated-compression framework with the gate, rather than isolating repeated-compression training alone. The controlled comparison available across Tables 2 and 3 shows that repeated training without the gate changes Video-MME, StreamingBench, and OVO-Bench by +1.26, −0.36, and +0.06, respectively. Thus, while the results support the joint framework, it would be better to clarify that repeated-compression training alone does not consistently improve all three benchmarks.

Questions for authors
  1. Since MURL is query-independent and trained using a single full-video caption, how does the reward encourage preserving details absent from that caption but required by future questions? Could the authors clarify the provenance and selection of the 40K captions and provide either a question-type/caption-coverage analysis or a comparison with frame/segment-level, QA-based, or mixed supervision?

  2. Could the authors supplement the controlled Qwen3.5 comparisons in Table 1 with the efficiency metrics from Table 7 and report hardware details and end-to-end query latency? The measurement should ideally include visual encoding, repeated memory updates, VLM prefill, time-to-first-token, and answer generation.

  3. Given the substantial baseline adaptations described in Appendix A.4, could the authors clarify how information access and causal memory constraints are matched across methods? Results from simple FIFO or uniform-sampling baselines under exactly the same fixed-capacity streaming protocol would help determine whether the gains in Table 1 arise from the learned policy rather than differences in baseline adaptation.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ke2H

We thank the reviewer for their valuable time and constructive suggestions. Below, we respond to the comments in Weaknesses (W) and Questions (Q).

W1&Q1: Question About Caption Supervision and Transfer Across Question Types

We thank the reviewer for raising this issue. We use detailed captions from LLaVA-Video-178K to supervise the retention of objects, attributes, actions, and event relationships without knowing future queries. Our design assumes that a combination of tokens that enables the model to reconstruct a detailed video caption is also more likely to retain visual information transferable to diverse downstream questions.

To evaluate the transferability of our training scheme across question types, we compared Qwen3.5-9B Base and MURL across Video-MME’s 12 question categories on the same 2,700 questions with identical video coverage. Base averages 11,520 visual tokens, while MURL averages 8,166.8, as queries may arrive while new frames are still accumulating after compression.

Question Type# QuestionsQwen3.5-9B Base (%)MURL (%)
Action Reasoning28565.9664.91
Action Recognition31372.2070.61
Attribute Perception22280.1881.53
Counting26855.9754.10
Information Synopsis32382.6681.73
OCR13977.7077.70
Object Reasoning45467.6266.74
Object Recognition35474.0174.86
Spatial Perception5475.9374.07
Spatial Reasoning5683.9387.50
Temporal Perception5578.1881.82
Temporal Reasoning17761.0267.23
Overall2,70071.3071.30

MURL matches Base on OCR and retains strong performance on attribute perception, object recognition, and spatiotemporal reasoning, with modest declines in counting and action recognition. These results support the transfer of detailed caption supervision across question types. For the 40K training pairs, we sampled equally from academic and YouTube sources and equally across predefined duration bins within each source. We used the original caption annotations, excluded revised duplicates, deduplicated videos, and removed missing or undecodable videos and empty or overlength captions. Additional sampling filled underrepresented strata, yielding 40,000 valid video–caption pairs.

W2&Q2:Hardware Details and End-to-End Efficiency Under Matched Qwen3.5 Settings

We thank the reviewer for this suggestion. Table 7 measures the additional memory-maintenance overhead. In Table 1’s controlled comparisons, all methods share the same Qwen3.5 visual encoder, projector, and language model, compress tokens after visual encoding, and use the same visual token count per query. Under identical execution settings, visual encoding and VLM prefill therefore have comparable computational costs. The main additional costs come from memory maintenance and auxiliary scoring, including QueryStream’s extra CLIP encoding. This is why we report maintenance latency separately. We we still agree that module-level timing and end-to-end latency should be clearly distinguished. We additionally measured efficiency on 200 Video-MME videos using Qwen3.5-9B in BF16 on one NVIDIA H200, with one warm-up and three measured repetitions per video. All methods follow the same 11,520 → 5,760 token schedule, averaging 21.08 updates per video.

MethodVisual Encoding (s)Memory Updates (s)Prefill (s)Query TTFT (s)Answer Generation (s)End-to-End Replay (s)Throughput (frames/s)
MURL11.69541.1530.5162.8760.835126.22425.94
FluxMem11.57080.4470.5902.7740.882164.78515.21
QueryStream11.26169.0620.4742.4990.735152.81717.40

Memory Updates is cumulative. Query TTFT starts at query-input assembly with memory prepared. Answer Generation includes prefill, and End-to-End Replay includes decoding and all subsequent processing. Throughput covers visual encoding, position construction, and memory maintenance. These metrics overlap and should not be summed. We will add these results, clarify the timing boundaries, and verify configuration consistency with the accuracy evaluation.

AuthorsOpenReview ↗

Response to Reviewer ke2H (Part 2)

W3&Q3: Fairness of Baseline Adaptations Under Matched Information Access and Memory Constraints

We thank the reviewer for raising this point. We designed the Table 1 adaptations to preserve each baseline’s effectiveness while giving all methods the same video frames and the same visual token budget at each query.

For StreamForest, we retained its ToMe merging rule and matched MURL’s compression trigger and retention budget. To check whether omitting the hierarchy weakened the baseline, we also evaluated an adaptation that restores short- and long-term memory, event segmentation, and recursive event merging. The ToMe baseline used in Table 1 achieved 67.15% overall accuracy, compared with 65.29% for the hierarchical adaptation.

Method (Qwen3.5-9B)Short (%)Medium (%)Long (%)Overall (%)
StreamForest with hierarchical memory80.4463.6751.7865.29
Fixed-capacity ToMe without hierarchical memory81.6765.2254.5667.15
MURL (ours)81.6771.4460.7871.30

For QueryStream, omitting RTAR has no effect on response timing because the protocol fixes when to answer. QueryStream retains its question-aware token selection. Our FluxMem adaptation can revisit the full visual history at query time, giving it broader access than MURL. These results and design choices support a fair comparison without placing the adapted baselines at a disadvantage.

To further isolate the contribution of learned selection, we evaluated FIFO and recursive uniform sampling on Video-MME with Qwen3.5-9B. Both baselines follow MURL’s 1 fps input and compression schedule. When memory reaches the equivalent of 128 frames, it is compressed to the equivalent of 64. FIFO retains the most recent frames, while recursive uniform sampling selects frames at equal intervals from the current memory. Since Qwen3.5 encodes each pair of adjacent frames as one temporal slice, both baselines select complete two-frame groups and retain all their visual tokens. Selection is independent of the question, and discarded content cannot be recovered. This comparison holds capacity, compression timing, retention budget, and access to past content constant, so the methods differ only in their selection strategy.

MethodShort (%)Medium (%)Long (%)Overall (%)
Frame-level FIFO81.2259.4449.2263.30
Frame-level recursive uniform sampling81.3362.7852.5665.56
MURL (ours)81.6771.4460.7871.30

W4: Clarification of Contribution of Repeated-Compression Training

We thank the reviewer for pointing this out. We agree that repeated-compression training alone yields inconsistent gains. The consistent improvements come when it is paired with the history gate, which allows the model to use information about each retained token’s past importance across compression rounds.

In the revision, we will report the gate-disabled comparison explicitly and clarify that Table 3 shows the benefit of the joint design.

Reviewer_ke2HOpenReview ↗

Response to Authors' Rebuttal

Thank you for the detailed rebuttal and the additional experiments. The end-to-end measurements substantially address my concern about the incomplete efficiency evaluation. They support improvements in processing throughput and memory-maintenance cost, although query TTFT is not improved. The additional FIFO, recursive uniform-sampling, and hierarchical-memory comparisons also strengthen the empirical evidence and substantially alleviate my concerns about the baseline adaptations. I appreciate the clarification that the consistent gains arise from repeated-compression training combined with the history gate.

My main remaining concern is the relationship between caption supervision and the preservation of information needed by future questions. The question-type breakdown is useful: matching the Base model’s overall accuracy with fewer visual tokens provides encouraging evidence of transfer. However, this comparison does not directly establish whether MURL preserves details absent from the supervision caption. The category-level gains and declines also leave the source of this transfer uncertain.

A focused analysis separating questions whose required evidence is covered by the corresponding caption from those requiring additional details would help address this issue. Alternatively, examples tracing whether such details survive repeated compression would be informative. I would also encourage the revision to distinguish the demonstrated ability to retain useful information across question types from the broader claim of preserving information relevant to unknown future queries.

Reviewer_KULZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Given a fixed visual memory capacity, a small token selector is trained via reinforcement learning to ensure that, as video frames continuously arrive and memory is repeatedly compressed, the model retains information most useful for subsequent understanding.

Critical strengths

Multi-turn memory updating is adopted as the direct training objective. The training process simulates real-world conditions where historical data has already been compressed, while new information continues to arrive.

A "history gate" is employed to mitigate shifts in the scoring context caused by repeated compression; it preserves not only visual features but also prior assessments of importance.

Video-level captions are used to provide feedback for multi-turn selection. This approach eliminates the need for token-level importance annotations and requires training only an additional module with approximately 10 million parameters.

Critical weaknesses

17:12

  1. Missing important baselines. The paper lacks comparisons with SCORE, a closely related RL-based token compression method, and SimpleStream, a strong recent-frame baseline. These comparisons are needed to establish the benefits of learned recursive memory updates.
  2. Incomplete implementations of competing methods. The adapted baselines omit important components, such as StreamForest’s hierarchical event memory, and modify the original memory-update procedures. The results therefore do not establish superiority over the complete competing systems.
  3. Insufficient evaluation of long-term memory. Aggregate benchmark scores do not directly demonstrate preservation of distant evidence. Results should be broken down by evidence-to-query delay and number of compression rounds.
  4. Confounded ablation study. The single-compression baseline also disables the history gate, so the comparison does not isolate the contribution of repeated-compression training.
  5. Limited efficiency and statistical evidence. The paper reports memory-maintenance time rather than end-to-end latency. Some improvements are small, and confidence intervals or results across multiple training seeds are missing.
Questions for authors

How does MURL compare with SCORE and SimpleStream under the same backbone and token budget?

How do the baseline adaptations affect performance compared with their complete official implementations?

How does performance change as the evidence-to-query delay and number of compression rounds increase?

Does repeated-compression training still outperform single-compression training when the history gate is disabled in both settings?

What are the end-to-end latency and training costs, and are the reported improvements consistent across multiple random seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KULZ

We thank the reviewer for the recognition of our work and for the constructive feedback. In response to the weaknesses and questions you raised, we provide the following replies(Below we respond to the comments in Weaknesses (W) and Questions (Q)):

W1&Q1: Missing comparison with SCORE and SimpleStream under a Matched Token Budget.

We thank the reviewer for this question. We reproduced the method described in the SCORE paper and applied it within the same memory-maintenance procedure used by MURL. We evaluated all three methods on Video-MME with the same backbone and token budget. For SimpleStream, we retained the most recent KK frames that fit this budget. The results are shown below:

MethodShort (%)Medium (%)Long (%)Overall (%)
SimpleStream81.2259.4449.2263.30
SCORE81.3368.4457.0068.93
MURL (ours)81.6771.4460.7871.30

MURL outperforms SCORE and SimpleStream by 2.37 and 8.00 percentage points in overall accuracy, respectively. We will include this comparison in the revised manuscript.

W2&Q2: Effect of Baseline Adaptations

We thank the reviewer for pointing out this. Our comparison uses the same Qwen3.5 backbone, evaluation examples, and final visual token budget for each example to evaluate compression strategies under controlled conditions. Since it was difficult to preserve StreamForest’s original hierarchical memory updates while matching the visual token budget exactly for every example, we omitted this mechanism from the original comparison. In a supplementary experiment, we separately evaluated a StreamForest adaptation that retains short- and long-term memory, event segmentation, and recursive event-level merging. The results on Video-MME are shown below.The StreamForest adaptation with hierarchical memory used an average of 6,826 visual tokens per example in the final input. The original fixed-capacity ToMe adaptation and MURL each averaged 8,166.

MethodShort (%)Medium (%)Long (%)Overall (%)
StreamForest with hierarchical memory80.4463.6751.7865.29
Fixed-capacity ToMe without hierarchical memory81.6765.2254.5667.15
MURL (ours)81.6771.4460.7871.30

For QueryStream, we use the response times specified by the evaluation protocol. Since RTAR determines when to respond autonomously, it is unnecessary under this fixed-time protocol, and omitting it leaves response scheduling unchanged. Our FluxMem adaptation can revisit all historical visual tokens available up to the query time, whereas MURL relies solely on retained compressed information. FluxMem thus has more permissive access to historical information. These experiments therefore provide a fair comparison of compression strategies under explicit budget constraints.

W3&Q3: Clarification of Memory Retention Across Query Delays and Compression Rounds.

We thank the reviewer for this question. Current datasets do not provide exact evidence timestamps, preventing direct analysis of how evidence-to-query delay affects performance. We therefore randomly selected 700 OVO-Bench samples with at least 300 seconds of video remaining after the original query time. Since the evidence for these questions appears before the original query, postponing the query by 75, 150, 225, or 300 seconds increases the evidence-to-query delay by the corresponding amount. During this interval, MURL continued processing the video and updating its memory. We then measured accuracy at each delay to assess its impact on performance.

Added query delayMURL (%)
0 s57.14
75 s57.14
150 s57.43
225 s58.00
300 s57.43

Accuracy remained between 57.14% and 58.00% across all delays. This suggests that MURL retains earlier evidence through repeated memory updates. To examine the effect of compression rounds, we grouped Video-MME questions by the number of rounds MURL performed and compared its accuracy with the 128-frame baseline on the same questions.

MURL compression roundsQuestionsBase-128f (%)MURL (%)
089783.2881.72
1–21275.0083.33
3–536373.8372.45
6–1043568.9770.34
11–209364.5270.97
>2090060.1160.78
Overall2,70071.3071.30

Both methods generally scored lower on videos requiring more compression rounds, but the accuracy gap showed no consistent trend. In the zero-round group, the videos were short, and Base-128f sampled more frames at a higher effective frame rate, which likely explains its advantage. Overall, these results suggest that additional compression rounds have limited impact on final performance, indicating that MURL effectively preserves key information through repeated compression.

AuthorsOpenReview ↗

Response to Reviewer KULZ (Part 2)

W4&Q4: Clarification of Repeated-Compression Training with the History Gate Disabled

We thank the reviewer for pointing out this issue. Our existing ablation results show that, when the history gate is disabled in both settings, repeated-compression training provides only limited gains over single-compression training. We therefore emphasize its synergy with the history gate. The gate allows each compression step to reference the historical importance of retained tokens, while repeated-compression training teaches the model to use this information to mitigate information loss from successive compressions. Their joint effect forms the core of our method. We agree that the original comparison, which changed both the training procedure and the gate setting, cannot isolate the contribution of repeated-compression training. In the revision, we will present the existing results without the gate more clearly and attribute the gains to the joint design.

W5&Q5: End-to-End Efficiency, Training Cost, and Seed Robustness

We thank the reviewer for pointing this out. We report end-to-end latency and its components below (all times in seconds).

MethodVisual Encoding (s)Memory Updates (s)Prefill (s)Query TTFT (s)Answer Generation (s)End-to-End Replay (s)Throughput (frames/s)
MURL11.69541.1530.5162.8760.835126.22425.94
FluxMem11.57080.4470.5902.7740.882164.78515.21
QueryStream11.26169.0620.4742.4990.735152.81717.40

MURL achieves lower end-to-end replay latency and higher throughput than both baselines, primarily through faster memory updates.

Training cost. The measured run used eight NVIDIA H200 GPUs and took approximately 55.30 hours (442.43 GPU-hours), from W&B initialization to the final checkpoint write. This includes frozen-backbone forward passes for reward computation, logging, visualization, and checkpointing.

Seed robustness. The results reported in the paper are means over three independently trained runs with different random seeds, using identical training settings and checkpoint-selection criteria. The results demonstrate that the reported improvements are consistent across multiple random seeds. We will state this clear in the revision. All checkpoints are evaluated with greedy decoding, so these statistics capture training variability rather than stochastic decoding.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal

Dear Reviewer KULZ,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results to address your concerns, and incorporated the key updates into the revised manuscript. With the discussion period drawing to a close, we would be grateful if you could revisit our rebuttal when convenient and let us know whether any points need further clarification. We would welcome the opportunity to address any remaining concerns.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to AC and Reviewers

Dear Reviewers and AC,

Thank you for your time and constructive feedback. In response to your feedback, we have provided additional experiments in our responses to clarify our intended points and further support our conclusions, and incorporated key experiments into the revised manuscript. All changes are highlighted in blue. The main updates are:

  • Direct comparison with repeated training. We have added numerical results to Table 3 in Section 4 and corresponding discussion in Section 4.3.
  • Additional baseline comparisons. We have added comparisons with SimpleStream and SCORE in Appendix A.5.
  • End-to-end efficiency. We have included additional efficiency results in Appendix A.5.
  • Impact of caption supervision. We have added experiments and analysis examining how caption supervision affects model behavior in Appendix A.6.

We kindly invite you to review these updates and look forward to your further feedback during the discussion period.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.