- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how streaming VLMs can retain historical information with bounded memory while remaining responsive to current observations. It proposes MaLoW, a recurrent hypernetwork that encodes video history into fixed-size, dynamically generated LoRA weights. The authors also construct MaLoW-Instruct for training. Experiments across streaming video benchmarks, multiple backbones, and a VLA setting demonstrate improvements over several baselines.
Critical strengths
S1: The method explores a useful alternative to representing historical information as embeddings in the input context.
Encoding history into dynamically generated LoRA weights bounds the context occupied by historical observations and avoids repeatedly processing the full history. This design addresses the competition between historical memory and current observations. Improvements across multiple backbones and tasks provide evidence that parameter-based memory is a promising direction.
Critical weaknesses
W1: A key concern is that the experiments do not distinguish the contributions of additional training data and the proposed memory mechanism.
The paper does not introduce an entirely new task, but the authors construct MaLoW-Instruct for additional training, while the compared methods are not reported with corresponding training on the same dataset. Therefore, the improvements may also be partly attributable to additional task supervision and cannot yet be clearly attributed to the proposed parameter-based memory mechanism. The new dataset can itself be a contribution, but establishing methodological advantages requires comparisons using the same training data.
W2: The results suggest that MaLoW is effective, while leaving some questions about its benefits across different video lengths.
For relatively short streams (OVO-bench), full-context inference remains feasible, and MaLoW does not consistently improve historical recall. For example, in Table 1, MOSS-VL-Realtime-11B achieves a Backward average of 72.5, compared with 71.7 for its MaLoW counterpart. For longer streams (StreamArena), Table 3 shows gains over the compared methods. However, MaLoW still suffers substantially degradation as history accumulates. The authors should further discuss these limitations, whether dynamic memory management or historical retrieval is needed, and evaluate longer settings to clarify the method’s practical scope.
Questions for authors
Q1: What is the fundamental advantage of encoding history into LoRA weights rather than memory embeddings?
The appendix establishes feasibility, but under the current design, a small, fixed number of memory tokens may also carry the recurrent memory while keeping the input context bounded.
Q2: Beyond the hour-scale StreamArena HR evaluation, how does the method generalize to substantially longer, day-long or cross-day streams (such as EgoLifeQA)?
Q3: How does recall of the same historical facts change as additional video is processed?
Does the current design include mechanisms to preserve important details or recover information lost from the fixed-size state?
References:
8 comments
Response to Reviewer Jvjy (Part 2)
W2 and Q3: degradation as history accumulates and recall of the historical facts over diverse temporal range.
We respectfully note that performance degradation as history accumulates is inherent in compressing growing historical information into fixed-size memory, which is shared by other memory mechanisms with bounded capacity. Table 1 of the main paper demonstrates that MaLoW outperforms alternative memory mechanisms.
Furthermore, the StreamArena HR results in Table C show that MaLoW, like other baselines, experiences performance degradation as relevant information becomes more temporally distant (from L1 to L4). Nevertheless, MaLoW maintains a clear performance advantage, demonstrating the effectiveness of the proposed memory mechanism.
To further investigate this limitation, we select 100 questions from the EPM task of OVO-Bench and extend the original video clips by appending random video segments of varying durations. The results are presented below.
Table C: Accuracy (%) on a subset of OVO-Bench EPM with randomly appended video segments of varying durations (seconds).
| Method / Appended duration (s) | 0 | 360 | 720 | 1080 | 1440 | 1800 | 2160 | 2520 | 2880 | 3240 | 3600 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MaLoW (Qwen3.5-9B) | 62.0 | 62.0 | 60.0 | 59.0 | 60.0 | 59.0 | 60.0 | 57.0 | 59.0 | 58.0 | 59.0 |
The results indicate a modest performance decline of 3 percentage points after appending up to 3,600 seconds of additional video, with accuracy remaining relatively stable across different history lengths.
Finally, as noted by the reviewer, MOSS-VL-Real-11B experiences performance drops on the OVO-Bench ASI and HLD tasks, reflecting a task-dependent trade-off. Nevertheless, MaLoW improves EPM accuracy from 60.6% to 62.0%, demonstrating its effectiveness on a core memory evaluation task. Moreover, it improves real-time accuracy from 75.2% to 77.4%, suggesting that MaLoW effectively balances memory retention and real-time perception.
W2 and Q3: discussion of information recovery and recall mechanism.
MaLoW employs a cosine-gated global semantic preservation branch to retain important visual information over time, serving as long-term memory, while the State-Space Module captures short-term dependencies. During LoRA generation, the converter retrieves memorized information from the hidden states and encodes it into LoRA weights. This design is conceptually similar to LSTMs, which maintain information in recurrent states and retrieve it when needed.
If the reviewer instead refers to explicit memory storage and retrieval mechanisms, such as retrieval-augmented generation (RAG), we emphasize that such approaches are particularly challenging for streaming video. With new visual observations arriving every second, continuously storing and retrieving information introduces substantial computational overhead, limiting practical real-time deployment. We therefore intentionally avoid explicit memory banks in our architecture.
Q1: What are the advantages of encoding memory into weights rather than retaining it in the context?
As discussed in the Introduction and shown in Figure 1, retaining historical information in the input context can introduce interference between long-term memory and real-time visual perception, degrading perception performance.
To further investigate, we conduct an ablation study using Qwen3.5-9B (Fixed Memory Tokens). Like MaLoW, this variant processes the most recent 2 seconds of video and incrementally integrates historical observations into an evolving hidden state. However, following Flash-VStream, it projects the hidden state into 16 memory tokens provided to the backbone as input context, rather than converting it into LoRA weights. We jointly train the memory module and Qwen3.5-9B backbone on MaLoW-Instruct using the training schedule in Section 4.1. We also evaluate a frozen-backbone variant, denoted as Qwen3.5-9B (Fixed Memory Tokens, Frozen Base). The results are presented in Table A above.
On OVO-Bench, the frozen-backbone variant reduces real-time perception accuracy from 75.5% to 70.2%, indicating interference from memory tokens. Joint fine-tuning improves accuracy to 72.6%, exceeding Qwen3.5-9B (Fine-tuned) by 0.9 percentage points, but still falling below the last-2s baselines without and with fine-tuning (75.5% and 74.2%, respectively). These results suggest that compressing historical information into fixed memory tokens does not fully resolve interference between memory and real-time perception. In contrast, MaLoW achieves 76.2% real-time perception accuracy while improving backward tracing accuracy to 66.4%, demonstrating that encoding memory into weights more effectively preserves real-time perception while leveraging historical information. These findings further validate the effectiveness of MaLoW's weight-based memory design.
Response to Authors
Thank you for the detailed response and additional experiments. The matched-data baselines, memory-token comparison, EgoLifeQA evaluation, and retention experiment substantially address several of my concerns. However, the following questions remain:
W1: The matched-data comparison shows that MaLoW provides additional gains on OVO-Bench. However, on StreamingBench-R, MaLoW and the baseline fine-tuned on the same data both achieve 82.3, compared with 81.9 before fine-tuning. Thus, the improvement on this benchmark is fully matched by additional fine-tuning, with no further gain from the proposed memory mechanism.
Q1: Thank you for adding the fixed-memory-token baseline. However, a total budget of 16 memory tokens appears rather restrictive. In §3.4 of Flash-VStream, each temporal memory slot contains 16 tokens, with 25 temporal slots, while the entire STAR memory has a total budget of 681 tokens. In contrast, the rebuttal baseline compresses the entire historical state into only 16 tokens. Please justify this choice and evaluate larger token budgets to determine whether the observed gap reflects an advantage of the LoRA interface or a bottleneck caused by an overly restrictive token-memory budget.
Response to Reviewer Jvjy (Part 1)
We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.
W1: Disentangle the effects of the training dataset MaLoW-Instruct and the proposed module MaLoW.
To isolate the contribution of MaLoW’s memory architecture, we conduct additional experiments on a matched training set using two baselines: Qwen3.5-9B (Fine-tuned), which retains historical observations in the input context, and Qwen3.5-9B+SimpleStream (Last 2s, Fine-tuned), which uses only the most recent two seconds. Both are fine-tuned on MaLoW-Instruct using rank-8 LoRA and the same training schedule as MaLoW (§4.1). These baselines enable controlled comparisons of encoding history in dynamically generated LoRA weights versus retaining it in the input context as memory or using only recent observations. The result is shown in Table A below.
Fine-tuning improves the full-context baseline’s OVO-Bench overall accuracy from 64.8 to 68.0, but its real-time perception remains below the local-window baselines, indicating persistent interference between historical memory and current perception. Under the same training setup, MaLoW improves real-time perception from 71.7 to 76.2 and backward tracing from 64.4 to 66.4, achieving 71.3 overall accuracy. These gains demonstrate the effectiveness of encoding historical memory in weights while preserving current perception. Notably, the last-2s fine-tuned baseline’s backward-tracing gains are largely driven by HLD, where all ground-truth answers are “Unable to answer,” rather than improved historical memory.
Finally, we want to reiterate that MaLoW-Instruct is also a separate contribution of this work and will be publicly released. We will clarify the training-data differences in the original comparisons and include this controlled experiment in the revised manuscript.
Table A: The effect of the Dataset and Comparison of context-based and weight-based memory representations.
OVO-Bench columns are grouped into Real-Time Visual Perception (RT) and Backward Tracing (BT), followed by the overall average.
W2 and Q2: Evaluation on the EgoLifeQA.
As requested, we provide additional evaluation of MaLoW (Qwen3.5-9B) on the Jake’s 500 questions of EgoLifeQA. The results are presented in Table B below. MaLoW (Qwen3.5-9B) achieves the best performance on EntityLog, EventRecall, and HabitInsight, and surpasses VST-7B, AURA-8B, and MOSS-VL-Realtime-11B by 1.7%, 2.8%, and 3.1% on average. Further demonstrating the effectiveness of the MaLoW.
Table B: The Performance on Jake’s 500 questions of EgoLifeQA. EL: EntityLog, RM: RelationMap, ER: EventRecall, TM: TaskMaster, HI: HabitInsight.