acceptodds
Under review as a conference paper at ICLR 2027

Memory as Weights: Internalizing Long-Term History for Streaming Videos

Abstract

Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context. We propose MaLoW, a streaming memory framework that instead internalizes historical context into evolving lightweight parameter updates. MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream. In doing so, MaLoW represents history as weights, while keeping the current observation as input. To support training, we further construct MaLoW-Instruct, a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos. We conduct extensive experiments to demonstrate that MaLoW achieves strong performance on OVO-Bench, StreamingBench, and StreamArena. On the episodic memory task of OVO-Bench, MaLoW outperforms prior approaches by 3.3%–6.7% across different video backbones while incurring only modest latency overhead. Beyond streaming video understanding, MaLoW also generalizes to vision-language-action models, improving the action success rate on Simpler WidowX by 9.3%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Vision Large Language Model, Histroy Internalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JvjyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how streaming VLMs can retain historical information with bounded memory while remaining responsive to current observations. It proposes MaLoW, a recurrent hypernetwork that encodes video history into fixed-size, dynamically generated LoRA weights. The authors also construct MaLoW-Instruct for training. Experiments across streaming video benchmarks, multiple backbones, and a VLA setting demonstrate improvements over several baselines.

Critical strengths

S1: The method explores a useful alternative to representing historical information as embeddings in the input context.

Encoding history into dynamically generated LoRA weights bounds the context occupied by historical observations and avoids repeatedly processing the full history. This design addresses the competition between historical memory and current observations. Improvements across multiple backbones and tasks provide evidence that parameter-based memory is a promising direction.

Critical weaknesses

W1: A key concern is that the experiments do not distinguish the contributions of additional training data and the proposed memory mechanism.

The paper does not introduce an entirely new task, but the authors construct MaLoW-Instruct for additional training, while the compared methods are not reported with corresponding training on the same dataset. Therefore, the improvements may also be partly attributable to additional task supervision and cannot yet be clearly attributed to the proposed parameter-based memory mechanism. The new dataset can itself be a contribution, but establishing methodological advantages requires comparisons using the same training data.

W2: The results suggest that MaLoW is effective, while leaving some questions about its benefits across different video lengths.

For relatively short streams (OVO-bench), full-context inference remains feasible, and MaLoW does not consistently improve historical recall. For example, in Table 1, MOSS-VL-Realtime-11B achieves a Backward average of 72.5, compared with 71.7 for its MaLoW counterpart. For longer streams (StreamArena), Table 3 shows gains over the compared methods. However, MaLoW still suffers substantially degradation as history accumulates. The authors should further discuss these limitations, whether dynamic memory management or historical retrieval is needed, and evaluate longer settings to clarify the method’s practical scope.

Questions for authors

Q1: What is the fundamental advantage of encoding history into LoRA weights rather than memory embeddings?

The appendix establishes feasibility, but under the current design, a small, fixed number of memory tokens may also carry the recurrent memory while keeping the input context bounded.

Q2: Beyond the hour-scale StreamArena HR evaluation, how does the method generalize to substantially longer, day-long or cross-day streams (such as EgoLifeQA)?

Q3: How does recall of the same historical facts change as additional video is processed?

Does the current design include mechanisms to preserve important details or recover information lost from the fixed-size state?

References:

[R1] EgoLife: Towards Egocentric Life Assistant

8 comments
AuthorsOpenReview ↗

Response to Reviewer Jvjy (Part 1)

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.

W1: Disentangle the effects of the training dataset MaLoW-Instruct and the proposed module MaLoW.

To isolate the contribution of MaLoW’s memory architecture, we conduct additional experiments on a matched training set using two baselines: Qwen3.5-9B (Fine-tuned), which retains historical observations in the input context, and Qwen3.5-9B+SimpleStream (Last 2s, Fine-tuned), which uses only the most recent two seconds. Both are fine-tuned on MaLoW-Instruct using rank-8 LoRA and the same training schedule as MaLoW (§4.1). These baselines enable controlled comparisons of encoding history in dynamically generated LoRA weights versus retaining it in the input context as memory or using only recent observations. The result is shown in Table A below.

Fine-tuning improves the full-context baseline’s OVO-Bench overall accuracy from 64.8 to 68.0, but its real-time perception remains below the local-window baselines, indicating persistent interference between historical memory and current perception. Under the same training setup, MaLoW improves real-time perception from 71.7 to 76.2 and backward tracing from 64.4 to 66.4, achieving 71.3 overall accuracy. These gains demonstrate the effectiveness of encoding historical memory in weights while preserving current perception. Notably, the last-2s fine-tuned baseline’s backward-tracing gains are largely driven by HLD, where all ground-truth answers are “Unable to answer,” rather than improved historical memory.

Finally, we want to reiterate that MaLoW-Instruct is also a separate contribution of this work and will be publicly released. We will clarify the training-data differences in the original comparisons and include this controlled experiment in the revised manuscript.

Table A: The effect of the Dataset and Comparison of context-based and weight-based memory representations.

OVO-Bench columns are grouped into Real-Time Visual Perception (RT) and Backward Tracing (BT), followed by the overall average.

Model#FramesStreaming Bench (R)RT: OCRRT: ACRRT: ATRRT: STURT: FPDRT: OJRRT: Avg.BT: EPMBT: ASIBT: HLDBT: Avg.OVO: Avg.
Qwen3.5-9B+SimpleStream (last 2s)282.086.671.678.460.180.276.175.556.954.741.951.263.3
Qwen3.5-9B+SimpleStream (last 2s, Fine-tuned)282.486.669.778.457.978.274.574.258.958.169.962.368.3
Qwen3.5-9B1 fps81.983.962.470.746.176.270.668.361.662.859.161.264.8
Qwen3.5-9B (Fine-tuned)1 fps82.383.966.175.953.480.270.771.762.064.067.164.468.0
Qwen3.5-9B (Fixed Memory Tokens, Frozen Base)1 fps81.286.657.871.655.176.273.970.260.357.452.256.663.4
Qwen3.5-9B (Fixed Memory Tokens)1 fps82.186.667.975.058.474.373.472.660.661.567.763.367.9
MaLoW (Qwen3.5-9B)1 fps82.387.966.181.060.184.277.776.263.666.269.466.471.3

W2 and Q2: Evaluation on the EgoLifeQA.

As requested, we provide additional evaluation of MaLoW (Qwen3.5-9B) on the Jake’s 500 questions of EgoLifeQA. The results are presented in Table B below. MaLoW (Qwen3.5-9B) achieves the best performance on EntityLog, EventRecall, and HabitInsight, and surpasses VST-7B, AURA-8B, and MOSS-VL-Realtime-11B by 1.7%, 2.8%, and 3.1% on average. Further demonstrating the effectiveness of the MaLoW.

Table B: The Performance on Jake’s 500 questions of EgoLifeQA. EL: EntityLog, RM: RelationMap, ER: EventRecall, TM: TaskMaster, HI: HabitInsight.

MethodELRMERTMHIAvg.
VideoChat3-4B24.814.421.46.416.416.7
MOSS-VL-Realtime-11B33.628.834.930.237.733.0
AURA-8B29.632.040.330.234.433.3
VST-7B34.425.638.136.237.734.4
MaLoW (Qwen3.5-9B)37.628.840.731.841.636.1
AuthorsOpenReview ↗

Response to Reviewer Jvjy (Part 2)

W2 and Q3: degradation as history accumulates and recall of the historical facts over diverse temporal range.

We respectfully note that performance degradation as history accumulates is inherent in compressing growing historical information into fixed-size memory, which is shared by other memory mechanisms with bounded capacity. Table 1 of the main paper demonstrates that MaLoW outperforms alternative memory mechanisms.

Furthermore, the StreamArena HR results in Table C show that MaLoW, like other baselines, experiences performance degradation as relevant information becomes more temporally distant (from L1 to L4). Nevertheless, MaLoW maintains a clear performance advantage, demonstrating the effectiveness of the proposed memory mechanism.

To further investigate this limitation, we select 100 questions from the EPM task of OVO-Bench and extend the original video clips by appending random video segments of varying durations. The results are presented below.

Table C: Accuracy (%) on a subset of OVO-Bench EPM with randomly appended video segments of varying durations (seconds).

Method / Appended duration (s)036072010801440180021602520288032403600
MaLoW (Qwen3.5-9B)62.062.060.059.060.059.060.057.059.058.059.0

The results indicate a modest performance decline of 3 percentage points after appending up to 3,600 seconds of additional video, with accuracy remaining relatively stable across different history lengths.

Finally, as noted by the reviewer, MOSS-VL-Real-11B experiences performance drops on the OVO-Bench ASI and HLD tasks, reflecting a task-dependent trade-off. Nevertheless, MaLoW improves EPM accuracy from 60.6% to 62.0%, demonstrating its effectiveness on a core memory evaluation task. Moreover, it improves real-time accuracy from 75.2% to 77.4%, suggesting that MaLoW effectively balances memory retention and real-time perception.

W2 and Q3: discussion of information recovery and recall mechanism.

MaLoW employs a cosine-gated global semantic preservation branch to retain important visual information over time, serving as long-term memory, while the State-Space Module captures short-term dependencies. During LoRA generation, the converter retrieves memorized information from the hidden states and encodes it into LoRA weights. This design is conceptually similar to LSTMs, which maintain information in recurrent states and retrieve it when needed.

If the reviewer instead refers to explicit memory storage and retrieval mechanisms, such as retrieval-augmented generation (RAG), we emphasize that such approaches are particularly challenging for streaming video. With new visual observations arriving every second, continuously storing and retrieving information introduces substantial computational overhead, limiting practical real-time deployment. We therefore intentionally avoid explicit memory banks in our architecture.

Q1: What are the advantages of encoding memory into weights rather than retaining it in the context?

As discussed in the Introduction and shown in Figure 1, retaining historical information in the input context can introduce interference between long-term memory and real-time visual perception, degrading perception performance.

To further investigate, we conduct an ablation study using Qwen3.5-9B (Fixed Memory Tokens). Like MaLoW, this variant processes the most recent 2 seconds of video and incrementally integrates historical observations into an evolving hidden state. However, following Flash-VStream, it projects the hidden state into 16 memory tokens provided to the backbone as input context, rather than converting it into LoRA weights. We jointly train the memory module and Qwen3.5-9B backbone on MaLoW-Instruct using the training schedule in Section 4.1. We also evaluate a frozen-backbone variant, denoted as Qwen3.5-9B (Fixed Memory Tokens, Frozen Base). The results are presented in Table A above.

On OVO-Bench, the frozen-backbone variant reduces real-time perception accuracy from 75.5% to 70.2%, indicating interference from memory tokens. Joint fine-tuning improves accuracy to 72.6%, exceeding Qwen3.5-9B (Fine-tuned) by 0.9 percentage points, but still falling below the last-2s baselines without and with fine-tuning (75.5% and 74.2%, respectively). These results suggest that compressing historical information into fixed memory tokens does not fully resolve interference between memory and real-time perception. In contrast, MaLoW achieves 76.2% real-time perception accuracy while improving backward tracing accuracy to 66.4%, demonstrating that encoding memory into weights more effectively preserves real-time perception while leveraging historical information. These findings further validate the effectiveness of MaLoW's weight-based memory design.

Reviewer_JvjyOpenReview ↗

Response to Authors

Thank you for the detailed response and additional experiments. The matched-data baselines, memory-token comparison, EgoLifeQA evaluation, and retention experiment substantially address several of my concerns. However, the following questions remain:

W1: The matched-data comparison shows that MaLoW provides additional gains on OVO-Bench. However, on StreamingBench-R, MaLoW and the baseline fine-tuned on the same data both achieve 82.3, compared with 81.9 before fine-tuning. Thus, the improvement on this benchmark is fully matched by additional fine-tuning, with no further gain from the proposed memory mechanism.

Q1: Thank you for adding the fixed-memory-token baseline. However, a total budget of 16 memory tokens appears rather restrictive. In §3.4 of Flash-VStream, each temporal memory slot contains 16 tokens, with 25 temporal slots, while the entire STAR memory has a total budget of 681 tokens. In contrast, the rebuttal baseline compresses the entire historical state into only 16 tokens. Please justify this choice and evaluate larger token budgets to determine whether the observed gap reflects an advantage of the LoRA interface or a bottleneck caused by an overly restrictive token-memory budget.

Reviewer_PnUtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MaLoW, a recurrent hypernetwork that encodes streaming video history into dynamically generated LoRA adapters for a frozen vision-language model, while retaining current observations as explicit input. It also constructs a streaming instruction dataset and evaluates the approach across multiple backbones, video understanding benchmarks, long-horizon retrospection, and robotic manipulation, with component ablations and efficiency analyses.

Critical strengths
  1. Useful system-level results. Figure 4 demonstrates a favorable memory-accuracy–latency trade-off against the evaluated current-window and full-history baselines, supporting the practical value of the implementation.

  2. Broad empirical coverage. The paper includes substantial training-data construction and evaluations across multiple backbones, long-horizon video tasks, and robotic manipulation, supplemented by component ablations and latency profiling.

Critical weaknesses
  1. The video-specific contribution is insufficiently established. Existing methods already encode context into LoRA parameters [1, 2]. The paper does not clearly identify which limitations of these methods require MaLoW's particular design, making the conceptual advance appear incremental.

  2. The comparisons do not isolate the benefit of the proposed mechanism. Comparisons with memory-equipped systems involve different backbones and training setups. Controlled comparisons against an alternative memory mechanism and a standard adaptation baseline trained on the same data are missing, leaving the contributions of architecture and additional training unclear.

  3. The distinctive components need more targeted validation. The ablations in Tables 11–12 show some improvements, but do not report ASI or long-gap recall. Their support for the intended temporal and long-term memory benefits remains limited.

[1] Chen et al. Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass. ICLR, 2025.

[2] Charakorn et al. Doc-to-LoRA: Learning to Instantly Internalize Contexts. ICML, 2026.

Questions for authors
  1. What specific limitation of existing LoRA-based memory methods does MaLoW address in streaming video, and which experiment demonstrates this?

  2. Do the SSM branch and visual gate improve temporal reasoning and long-gap recall? Targeted ablations on ASI and StreamArena's temporal-gap groups would help clarify their roles.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PnUt (Part 1)

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables/Figures A, B, etc. refer to those in the rebuttal.

W1 and Q1: The novelty and necessity of the MaLoW.

Although [1, 2] also explore encoding context into generated adapter weights, they are not tailored to the needs of streaming video (latency requirements, redundancy control, causal relationship modeling). We will add these papers to the paper and revise the discussion accordingly.

Conceptually, MaLoW is motivated by the observation that retaining historical context of streaming video can interfere with real-time perception, as illustrated in Figure 1. To address this challenge, MaLoW proposes to internalize historical information into model weights rather than retaining it in the context window and only taking the current observation as input.

Technically, MaLoW is tailored for streaming video setting, considering the latency requirement, redundancy of the video signal, casual relationship of the video signal. 1. MaLoW proposes to use the features of the vision encoder instead of the features after the LLM as [1, 2], reducing the inference time. 2. MaLoW proposes to use the learnable query and aggregator to reduce the spatial redundancy and further use cosine gating to reduce the temporal redundancy of the video. 3. MaLoW proposes to use SSM and cosine gating for modeling the causal relationship and overall semantics of the video history.

Performance-wise, the latency-and-memory trade-off in Figure 4 in the main paper, MaLoW obtains 63.6 on the OVO-bench EPM task with 153ms latency, demonstrating the effectiveness of redundancy reduction and structural design. Tables 11 and 12 in the appendix directly show the effectiveness of MaLoW's memory design for streaming video, combining the cos-gated sum and the SSM yields the best result 68.5 on OVO-Bench. Finally, the results in Table 1, Table 16, and Table 3 demonstrate MaLoW's superior performance over the baselines on OVO-Bench, StreamingBench, and StreamArena. These results collectively support MaLoW's design for streaming Video.

W2: Disentangle the effects of the training dataset MaLoW-Instruct and the proposed module MaLoW.

To isolate the contribution of MaLoW’s memory architecture, we conduct additional experiments (Table A below) on a matched training set using two baselines: Qwen3.5-9B (Fine-tuned), which retains historical observations in the input context, and Qwen3.5-9B+SimpleStream (Last 2s, Fine-tuned), which uses only the most recent two seconds. Both are fine-tuned on MaLoW-Instruct using rank-8 LoRA and the same training schedule as MaLoW (§4.1). These baselines enable controlled comparisons of encoding history in dynamically generated LoRA weights versus retaining it in the input context as memory or using only recent observations.

Fine-tuning improves the full-context baseline’s OVO-Bench overall accuracy from 64.8 to 68.0, but its real-time perception remains below the local-window baselines, indicating persistent interference between historical memory and current perception.

Under the same training setup, MaLoW improves real-time perception from 71.7 to 76.2 and backward tracing from 64.4 to 66.4, achieving 71.3 overall accuracy. These gains demonstrate the effectiveness of encoding historical memory in weights while preserving current perception.

Notably, the last-2s fine-tuned baseline’s backward-tracing gains are largely driven by HLD, where all ground-truth answers are “Unable to answer,” rather than improved historical memory.

Finally, we want to reiterate that MaLoW-Instruct is also a separate contribution of this work and will be publicly released. We will clarify the training-data differences in the original comparisons and include this controlled experiment in the revised manuscript.

Table A: The effect of the Dataset. The OCR–Avg. (Real-Time) columns are OVO-Bench real-time visual perception; EPM–Avg. (Backward) are OVO-Bench backward tracing. The rightmost Avg. is the OVO-Bench overall average.

Model#FramesStreamingBench (R)OCRACRATRSTUFPDOJRAvg. (Real-Time)EPMASIHLDAvg. (Backward)Avg. (Overall)
Qwen3.5-9B+SimpleStream (last 2s)282.086.671.678.460.180.276.175.556.954.741.951.263.3
Qwen3.5-9B+SimpleStream (last 2s, Fine-tuned)282.486.669.778.457.978.274.574.258.958.169.962.368.3
Qwen3.5-9B1 fps81.983.962.470.746.176.270.668.361.662.859.161.264.8
Qwen3.5-9B (Fine-tuned)1 fps82.383.966.175.953.480.270.771.762.064.067.164.468.0
MaLoW (Qwen3.5-9B)1 fps82.387.966.181.060.184.277.776.263.666.269.466.471.3
AuthorsOpenReview ↗

Response to Reviewer PnUt (Part 2)

W3 and Q2: Additional Ablation

We extend the component ablations with OVO-Bench Action Sequence Identification (ASI) and StreamArena Historical Retrospection (HR), including its four evidence-to-query gap groups: L1 (<5 min), L2 (5–15 min), L3 (15–30 min), and L4 (>30 min).

SSM and global semantic aggregation. Table B shows that adding SSM to the summation branch improves ASI accuracy from 62.2 to 64.2 (+2.0 percentage points) and the overall HR score from 21.3 to 22.0 (+0.7 points). For longer temporal gaps, L3 improves from 16.8 to 19.6 (+2.8 points), and L4 from 17.0 to 17.6 (+0.6 points). These results support complementary contributions from the two branches, with their combination achieving the strongest ASI and overall HR performance.

Visual gating. With the SSM branch retained, Table C isolates the aggregation strategy. Compared with plain summation, gated summation improves ASI from 60.1 to 64.2 (+4.1 percentage points) and HR from 20.2 to 22.8 (+2.6 points), with gains of 4.9 points on L3 and 1.7 points on L4. It also outperforms weighted summation on ASI and every HR gap group, including gains of 2.2 and 0.6 points on L3 and L4.

Together, these targeted ablations support the contribution of SSM within the combined architecture and the benefit of visual gating for action-sequence reasoning and long-gap historical recall.

Table B: Ablation study on the causal relationship (SSM) and global semantics.

VariantOVO-Bench (ASI)StreamArena (HR)L1L2L3L4
w/ sum62.221.322.725.816.817.0
w/ SSM59.519.723.223.614.114.8
w/ SSM + sum64.222.023.225.219.617.6

Table C: Ablation study on different global semantics aggregation strategies.

VariantOVO-Bench (ASI)StreamArena (HR)L1L2L3L4
sum60.120.221.523.916.315.9
weighted sum63.521.121.024.519.017.0
gated sum64.222.823.826.121.217.6

Thank you for the clarifications and additional experiments. The fine-tuning baselines substantially address my concern about the contribution of additional training, and the new ASI and long-gap recall ablations provide stronger support for the proposed components.

My main remaining concern is the contribution beyond existing LoRA-based memory methods. While the response clarifies the design motivations, the methodological advance and its benefit over related parametric-memory approaches remain insufficiently established. Taking the improved experimental evidence and this remaining concern into account, I maintain my weak-reject recommendation.

Thanks for the comment. We further clarify the concpetual and methodological difference between our approach and [1,2]. We acknowledge that Doc-to-LoRA [1] and GenerativeAdapter [2] share the general idea of internalizing historical context into model parameters through a learned adapter generator. However, MaLoW differs substantially in its memory architecture and problem formulation, addressing challenges specific to streaming video. Below, we provide a detailed discussion of these distinctions. We will incorporate a condensed version into the main paper.

Distinction from Doc-to-LoRA

Memory size vs context length. Although Doc-to-LoRA supports long-context processing through chunking, it independently generates LoRA adapters for individual chunks and combines them through rank-wise concatenation, resulting in a memory representation whose rank grows linearly with the number of chunks. In contrast, MaLoW employs a recurrent hypernetwork to continuously consolidate incoming observations into a fixed-size hidden state and LoRA representation, maintaining constant memory capacity regardless of stream duration.

Temporal dependencies among chunks. Doc-to-LoRA's independent chunk aggregation does not explicitly model temporal dependencies across observations. In contrast, MaLoW employs a state-space model to capture temporal evolution, together with gated semantic accumulation to preserve long-term information while suppressing redundant visual observations. These designs address key challenges in streaming video, where event ordering, temporal transitions, and substantial visual redundancy are critical to understanding the observed activities.

Distinction from GenerativeAdapter

Explicit temporal modeling. Unlike GenerativeAdapter's additive aggregation of contextual features, MaLoW introduces a recurrent state-space model to explicitly capture temporal dependencies and evolving relationships between observations, which are critical for understanding dynamic events in streaming video.

Temporal redundancy. MaLoW incorporates similarity-based gating to selectively accumulate historical information while suppressing redundant visual observations. This addresses a characteristic challenge of continuous video streams that is not considered in GenerativeAdapter.

Complementary memory representations. MaLoW combines recurrent temporal modeling with long-term semantic accumulation, enabling complementary representations of temporal dynamics and persistent historical context. In contrast, GenerativeAdapter relies on a single additive aggregation mechanism.

Remak

Together, these distinctions establish MaLoW as a recurrent, video-specific memory consolidation framework. Rather than directly applying prior approaches, MaLoW significantly extends the broader paradigm of parameter-based context internalization to continuous multimodal perception, with architectural designs tailored to the temporal structure and redundancy of streaming video.

Reviewer_rTWJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript addresses video-question answering in a streaming setup. In order to store past observations, the paper proposes to learn and update LoRA weights for a pre-trained VLM. The LoRA weights are updated based on a recurrent State-Space Model and a simple aggregation of video features over time. The model is trained on a newly curated training dataset. The approach is evaluated on StreamingBench, OVO-Bench, SimplerEnv-Bridge, and StreamArena.

Critical strengths

The idea of using LoRA weights as memory is very interesting and novel in the context of video-question answering in a streaming setup. Using a recurrent state space model for updating the LoRA weights is also a reasonable approach. The idea and contribution of the paper are a clear strength of the paper.

The experimental evaluation shows that the approach generalizes to various backbones. Reporting results for VideoChat3, Qwen3.5, MOSS-VL-Real, and Gemma4 as well as models of different sizes (Table 4) demonstrates that the approach works very well for various VLMs. Showing results not only for video-question answering but also for predicting future robot actions is another convincing evidence of the usefulness of the proposed approach.

The ablation studies in the Appendix evaluate the most important aspects like LoRA rank or state space model vs. feature aggregation. It would be useful to move some of the ablation studies (Table 10 and 11) to the paper.

The training and evaluation code, trained model checkpoints, and processed MaLoW-Instruct data will be released to support reproducibility.

Critical weaknesses

W1 Fairness of comparisons:

The paper introduces a newly curated training dataset. While the dataset is an asset, it raises major concerns regarding the fairness of the comparison with state-of-the-art methods (Tables 1 and 16). It needs to be clarified if the methods used the same training data. The gain of the contribution (LoRA weights as memory) vs. the gain of the training data and training procedure are not sufficiently disentangled.

W2 Test-time adaptation/training:

The approach does basically test-time adaptation/training with LoRA. While the paragraph "Low-Rank Adaptation and Context Internalization" discusses some related work, the conceptually differences are not very clear. In particular, it is unclear how much or what is memorized and how much of the gain is due to adaptation.

W3 Hidden-State Update:

Eq. (6) accumulates the previous features without normalization or deleting old information. It only discards video clips that are very similar to the previous video clip. While s is normalized before predicting the LoRA weights, summing features over time has the negative impact that if the norm of s increases over time new information q has very little impact on s when the streaming video is long. While this limitation does not seem to be an issue for the used benchmarks, it might be a limitation for longer videos. This might also explain the benefit of using the state space model output o and the aggregation s since the features of s and o might be dominated by different parts of the video.

W4 Details:

The paper is well written and important details are in the Appendix, but more details should be added to the LoRA Decoding and Loss Function. In particular, the LoRA decoding needs to be described more in detail. For instance, the gating parameter α is not part of (7).

Questions for authors

Major:

Q1 Are the results in Tables 1 and 16 based on the same training data? If not, a 1:1 comparison is required for a subset of the methods, where each method is trained on the same training data and (if possible) same loss/training procedure.

Q2 Technically, the approach does test-time adaptation/training with LoRA, e.g., "LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models (Workshop on Test-Time Adaptation: Putting Updates to the Test! at ICML 2025). How does the proposed approach differ from test-time adaption with LoRA in a streaming setup? A comparison to a test-time training approach with LoRA would strengthen the evaluation. It would be also helpful if it could be quantified how much the performance gains come from adapting to the sequences and how much comes actually from memorization. See e.g.

"Understanding LoRA as Knowledge Memory: An Empirical Analysis" ICML 2026

"Training plug n’play knowledge modules with deep context distillation" COLM 2025

Q3 Please discuss/analyze the limitation of the accumulation s for long videos. Does s have a negative impact if the video has been already streamed for longer time and the current question is related to the more recent past (excluding the current observation)?

Q4 The memory aspect is also addressed for other streaming tasks like video narration, e.g., "Videollmonline: Online video large language model for streaming video" CVPR 2024 or "FlowNar: Scalable Streaming Narration for Long-Form Videos" ICML 2026. Reporting results on narration is not required, but an interesting future work. How does the memory handling for video narration differ from the memory handling for video-question answering?

Minor:

  • It would be useful to move some of the ablation studies (Table 10 and 11) to the paper.

  • Typo: "Unique Wideos" in Table 5.

  • (6) is not an LSTM. The LSTM reference in l.214 is misleading and should be removed.

  • "Using 100 deterministically sampled OVO-Bench questions": It would be good to provide the list of questions as part of the source code.

7 comments
AuthorsOpenReview ↗

Response to Reviewer rTWJ (Part 1)

We thank the reviewer for the constructive feedback. Throughout our responses, Tables/Figures 1, 2, etc. refer to those in the original paper, while Tables A, B, etc. refer to those in the rebuttal. We address the reviewer's questions below.

W1: Disentangle effect of training dataset MaLoW-Instruct and the proposed module MaLoW.

To isolate the contribution of MaLoW’s memory architecture, we conduct additional experiments on a matched training set using two baselines: Qwen3.5-9B (Fine-tuned), which retains historical observations in the input context, and Qwen3.5-9B+SimpleStream (Last 2s, Fine-tuned), which uses only the most recent two seconds. Both are fine-tuned on MaLoW-Instruct using rank-8 LoRA and the same training schedule as MaLoW (§4.1). These baselines enable controlled comparisons of encoding history in dynamically generated LoRA weights versus retaining it in the input context as memory or using only recent observations. The results are presented in Table A below.

Fine-tuning improves the full-context baseline’s OVO-Bench overall accuracy from 64.8 to 68.0, but its real-time perception remains below the local-window baselines, indicating persistent interference between historical memory and current perception. Under the same training setup, MaLoW improves real-time perception from 71.7 to 76.2 and backward tracing from 64.4 to 66.4, achieving 71.3 overall accuracy. These gains demonstrate the effectiveness of encoding historical memory in weights while preserving current perception. Notably, the last-2s fine-tuned baseline’s backward-tracing gains are largely driven by HLD, where all ground-truth answers are “Unable to answer,” rather than improved historical memory.

Finally, we want to reiterate that MaLoW-Instruct is also a separate contribution of this work and will be publicly released. We will clarify the training-data differences in the original comparisons and include this controlled experiment in the revised manuscript.

Table A: The effect of the Dataset. OVO-Bench columns are grouped into Real-Time Visual Perception (OCR, ACR, ATR, STU, FPD, OJR, RT Avg.), Backward Tracing (EPM, ASI, HLD, BT Avg.), and overall average (OVO Avg.).

Model# FramesStreamingBench (R)OCRACRATRSTUFPDOJRRT Avg.EPMASIHLDBT Avg.OVO Avg.
Qwen3.5-9B+SimpleStream (last 2s)282.086.671.678.460.180.276.175.556.954.741.951.263.3
Qwen3.5-9B+SimpleStream (last 2s, Fine-tuned)282.486.669.778.457.978.274.574.258.958.169.962.368.3
Qwen3.5-9B1 fps81.983.962.470.746.176.270.668.361.662.859.161.264.8
Qwen3.5-9B (Fine-tuned)1 fps82.383.966.175.953.480.270.771.762.064.067.164.468.0
MaLoW (Qwen3.5-9B)1 fps82.387.966.181.060.184.277.776.263.666.269.466.471.3

Q1: Training data in Table 1 and 16.

The methods in Tables 1 and 16 were trained on different datasets, and these comparisons primarily reflect overall system performance. We have provided additional results on matched training data. See our response to W1.

AuthorsOpenReview ↗

Response to Reviewer rTWJ (Part 2)

W2 and the Q2: Difference between test-time adaptation and memory internalization.

We clarify the conceptual differences and provide additional comparisons to a concurrent work on test-time adaptation in a streaming setup.

Distinction. MaLoW presents a test-time adaptation framework for streaming video understanding. It differs from prior approaches that leverage test-time training (e.g., LoRA-TTT) by avoiding costly backpropagation to update LoRA weights. Instead, MaLoW proposes a hypernetwork to directly generate LoRA weights from incoming video clips, and incrementally integrates these weights over time to encode historical information.

New results. As requested, we further compare MaLoW (Qwen3-VL-4B) with a concurrent work, StreamTTT-4B, which is a test-time training approach in Table B below. MaLoW (Qwen3-VL-4B) surpasses StreamTTT-4B by 3.9% on the Backward split of the OVO-Bench, which demonstrates the effectiveness of MaLoW over the test-time training approach.

Chen, Joya, Zeyun Zhong, and Mike Zheng Shou. "StreamTTT: Reconciling real-time perception and long-term memory in streaming VLMs." arXiv:2608.13416 (2026/09).

Related works. We acknowledge the previous works Understanding LoRA as Knowledge Memory and Training plug n’play knowledge modules with deep context distillation, which establish and investigate LoRA-based knowledge storage. We will add these papers to the discussion of the related work in the revised manuscript.

Table B: Compare MaLoW with the test-time training-based method. OVO-Bench columns are grouped into Real-Time Visual Perception (OCR, ACR, ATR, STU, FPD, OJR, RT Avg.), Backward Tracing (EPM, ASI, HLD, BT Avg.), and overall average (OVO Avg.).

Model# FramesStreamingBench (R)OCRACRATRSTUFPDOJRRT Avg.EPMASIHLDBT Avg.OVO Avg.
StreamTTT-4B (Chen et al., 2026)2 fps81.391.380.780.256.282.277.778.160.958.160.859.969.0
MaLoW (Qwen3-VL-4B)1 fps84.890.669.775.064.074.374.574.762.067.661.863.869.3

W2 and the Q2: Quantify the effect of adaptation and memorization.

Adaptation serves as the key mechanism for memorization in MaLoW. This is because MaLoW internalizes memory by updating model parameters, and thus “adapting” a pretrained model to incoming video streams. In this context, the two concepts are fundamentally coupled in our design. We will clarify this relationship in the final manuscript. We are happy to have a further discussion.

W3 and the Q3: The design of the Global Semantic Branch in Memory:

The global semantic preservation branch is conceptually similar to the persistent cell state in LSTMs. This additive branch is designed to preserve global semantic information. To complement it, we employ a state-space model (SSM) to capture short-term temporal dependencies. The ablation results in Table 11 of the appendix demonstrate that combining both branches yields better performance than using either alone.

As requested, we conduct an additional experiment examining how prolonged video history affects the retrieval of information from the recent past. Specifically, we select 100 questions from the OVO-Bench Episodic Memory (EPM) task and prepend randomly sampled videos of up to one hour to the original video clips, simulating increasingly long historical contexts while keeping the relevant memory clues in the more recent portion of the stream. We compare two configurations with MaLoW (Qwen3.5-9B): (1) using only the global semantic preservation branch and (2) combining the global semantic preservation and SSM branches. The results are presented in the table below:

Table C: Accuracy (%) in a subset of the EPM tasks of OVO-Bench by prepending random video. Duration in seconds. GSB: Global Semantic Branch.

Method036072010801440180021602520288032403600
W/ GSB62.062.060.061.060.059.060.059.059.060.059.0
W/ GSB + SSM62.062.062.061.062.062.062.062.061.062.062.0

As shown in Table C above, using only the Global Semantic Branch, increasing the length of the preceding history slightly degrades memory performance, matching the intuition behind the global semantic branch. However, combining the SSM branch with global semantic preservation mitigates this degradation, demonstrating the benefit of modeling both local temporal dependencies and global historical semantics.

W4: Implementation details in the paper.

We will update Equation 7 and will move details on LoRA decoding and the training loss to the main paper, as space permits. These details are currently provided in the sections The Converter and Training Details in Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer rTWJ (Part 3)

Q4: Applying MaLoW on Narration.

This is an interesting direction! Both streaming narration and video QA require incremental memory but impose different demands. QA must preserve visual evidence for future questions whose relevance may be unknown at observation time. In contrast, narration requires temporal continuity, tracking previously generated descriptions, deciding when to speak, and mitigating accumulated errors.

MaLoW internalizes historical information into evolving LoRA weights while processing current observations as input, making it potentially applicable to both tasks. Its existing ability to encode timestamp-aligned text alongside visual observations provides a foundation for streaming narration. However, this extension requires additional dataset curation and investigation, which we leave for future work.

Minor:

We will move some ablation into the main paper in the final manuscript. The typos are fixed in the revised manuscript. We will release the list of the OVO-Bench questions as part of the source code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.