acceptodds
Withdrawn from ICLR 2027

A Dual-memory Framework for Language-driven Action Localization in Streaming Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-driven action localization in streaming video aims to temporally ground natural language descriptions in an ongoing video stream, using only past and current frames. In contrast to offline settings with full video access, the streaming scenario lacks future context, leading to substantial uncertainty when recognizing partially observed actions. To address this issue, we draw inspiration from human cognition, where short-term experiences are gradually abstracted and integrated into long-term knowledge that helps interpret or anticipate actions under limited observation. We propose a dual-memory framework, which consists of two complementary memory modules: (1) an intra-memory module that encodes short-term temporal context from historical frames within the current video stream, capturing scene-specific static cues and scene-agnostic action cues, and (2) an inter-memory module that maintains long-term prototypical action representations distilled from previously observed videos. Importantly, the inter-memory is formed by integrating scene-agnostic action representations from intra-memory instances across multiple videos, reflecting how humans consolidate long-term knowledge from repeated short-term observations. During localization, the intra-memory provides the scene-specific reference, while the inter-memory retrieves plausible scene-agnostic action trends. By fusing them, our model synthesizes scene-specific future representations, compensating for the absence of future frames. Comprehensive experiments on several benchmark datasets demonstrate that our method consistently outperforms existing methods under the streaming setting, validating the effectiveness of our cognitively inspired dual-memory design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dual-memory, Language-driven Action Localization, Streaming Video

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wffeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes a dual-memory framework for retrieving video moments in streaming videos. It uses two modules: one for short-term context within the current video, and another for long-term action knowledge across videos. This helps compensate for missing future frames and improves temporal localization. Experiments on multiple benchmarks confirm its effectiveness.

Critical strengths
  1. The paper presents a clear motivation and a well-structured framework. The overall methodology is clearly explained and effectively illustrated, making the proposed approach easy to understand.
  2. The paper provides extensive experiments and ablation studies across multiple benchmarks and backbone models, demonstrating the effectiveness of the proposed method.
Critical weaknesses
  1. Memory-based temporal modeling and future anticipation have already been explored in streaming video moment retrieval (e.g., TSGSV [1] and HEM [2]). While the proposed Header–Slider structure offers a novel way to organize cross-video action knowledge, its distinct contribution over these closely related methods remains insufficiently established.
  2. In Eq. (10), current visual features and static memory are concatenated with learnable future tokens as queries for cross-attention. However, under standard cross-attention, future query outputs do not directly incorporate information from other query tokens. It is therefore unclear how retrieved action trends are effectively adapted to the current scene.
  3. While Table 11 shows that retrieving subsequent memory features improves localization, it does not verify whether these features accurately capture future action dynamics. Furthermore, the stage-wise analysis in Table 10 lacks baseline comparisons, making it difficult to assess if the method specifically enhances localization during early observation stages where future context is most limited.
  4. Table 7 indicates that Dual-Memory nearly doubles the parameter count (7.91M to 14.91M). Without a parameter-matched baseline (e.g., a larger backbone or simpler memory module with comparable capacity), it is unclear whether the performance gains stem from the proposed memory design or merely the increased model capacity.

[1] Gan T, Wang X, Sun Y, et al. Temporal sentence grounding in streaming videos[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4637-4646. [2] Zheng M, Peng Y, Sun B, et al. Hierarchical event memory for accurate and low-latency online video temporal grounding[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 21589-21599.

Questions for authors
  1. Clarify the core methodological novelty compared with existing memory-based approaches, particularly TSGSV and HEM, and highlight the unique contributions of the proposed dual-memory framework.
  2. Explain how Eq. (10) incorporates scene-specific static information and current visual features into the generated future tokens, particularly given the standard formulation of cross-attention.
  3. Provide further evidence that the retrieved future features accurately represent future action dynamics, rather than simply serving as additional contextual information.
  4. Clarify whether the proposed framework provides greater improvements during the early stages of action observation, where future information is most limited.
  5. Discuss the influence of increased model capacity and whether comparable improvements could be achieved using a parameter-matched backbone without the proposed dual-memory mechanism.
Reviewer_pGPGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a dual-memory framework to address the lack of future information in streaming action localization. Intra-memory compresses historical features from the current video to capture scene-specific static information and action dynamics. Inter-memory stores action prototypes learned across videos and uses a Header–Slider structure to retrieve relevant actions and their temporal stages, providing future action trends to assist localization of the current segment. The authors integrate the framework into several existing localization models and report consistent improvements. Ablation studies also support the utility of the memory components and the future-trend retrieval strategy.

Critical strengths
  1. The dual-memory framework is well motivated and yields consistent reported gains across multiple backbones. Intra-memory retains historical information from the current video, while Inter-memory provides action prototypes learned across videos. The Header–Slider structure performs action-semantic retrieval and temporal-stage matching to supply future trends under partial observation. In Table 4, adding Inter-memory further improves performance over Intra-memory alone, supporting the complementary roles of the two memories.

  2. The future-trend retrieval and cross-video transfer experiments provide useful, targeted evidence. Appendix B.3 compares four readout positions with fixed model parameters. After match achieves 35.82 [email protected], compared with 32.48–33.24 for the alternatives. Appendix B.4 freezes an Inter-memory trained on ActivityNet Caption and trains the remaining parameters on QVHighlights, outperforming a frozen random memory. These controls examine the roles of readout position and memory content more directly than simply removing a module.

Critical weaknesses
  1. The learning mechanism and temporal-stage alignment of the Header–Slider action prototypes remain unclear. Equation (5) selects a memory slot using argmax, while Eq. (11) only constrains Header orthogonality, leaving the training signal for semantic alignment unspecified. It is also unclear how the gated update in Eq. (9), the stop-gradient operation in Eq. (12), and optimizer updates work together. Moreover, Eqs. (2)–(3) do not specify a temporal ordering for Intra-memory tokens, yet Eq. (6) matches them to consecutive Slider positions. Smoothness does not establish chronological order. Appendix B.3 supports reading after the match, but does not directly validate the correspondence between Slider positions and action progress, leaving the temporal interpretation of the retrieved future trends uncertain.

  2. The role of current-segment information in stage matching and future feature synthesis remains unclear. Equation (6) matches the Slider using a dynamic action channel constructed only from the historical segment, without using action changes in the current segment. This may cause stage matching to lag when a key change has just occurred. Equation (10) concatenates the static channel, current features, and learnable pseudo-future tokens as queries, then retains only the outputs at the pseudo-future token positions. Under standard CrossAttn, conditional on a fixed retrieved trend and without additional interactions across query tokens, this concatenation does not make the retained outputs depend on the preceding static or current features. The subsequent use of current features by the base model does not establish that this synthesis step produces the scene-specific future representations described in the paper.

Questions for authors
  1. How are Header and Slider retrieval and updates trained? Please explain the semantic training signal for the Headers and how the gated update in Eq. (9), the stop-gradient operation sg in Eq. (12), and optimizer updates work together. How is cosine similarity aggregated over tokens in Eqs. (5) and (6)?

  2. How is temporal alignment established between Intra-memory tokens and Slider positions? Attention pooling does not explicitly specify a temporal ordering for the compressed tokens. What training mechanism gives these tokens a temporal organization compatible with consecutive Slider positions? Please provide examples or analysis showing how the matched positions change with actual action progress.

  3. How does Eq. (10) produce future representations specific to the current scene? Please clarify whether CrossAttn includes self-attention within the reference sequence or another fusion operation, and identify the computational path through which the static channel and current features influence the final future representation. Without such an operation, how does the result differ from using only the learnable pseudo-future tokens as queries?

  4. Why are action changes in the current segment not used for Slider stage matching? Please explain the rationale for using historical dynamics in Eq. (6), and whether the “future” read after the match is defined relative to the end of the historical segment or the current segment. How is a potential stage-matching lag handled when a key change has just occurred in the current segment?

  5. How is the scene-agnostic nature of the Inter-memory action prototypes validated? The dynamic action channel in Section 3.3.1 comes from temporal differences, which may still be affected by camera motion or background changes. Table 5 and Appendix B.4 support the usefulness of temporal differences and memory transfer. Could retrieval results for the same type of action across different scenes further assess the prototypes' sensitivity to scene changes?

Reviewer_UBW8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses language‑driven action localization for streaming videos, where only past and current frames are accessible and future frames are unavailable. Inspired by human cognitive memory, the authors propose a dual‑memory framework consisting of intra‑memory and inter‑memory modules. The intra‑memory compresses short‑term historical visual context into scene‑specific static and scene‑agnostic action representations. The inter‑memory maintains long‑term prototypical action patterns learned across different videos, enabling future‑trend synthesis to compensate for missing future frames. Extensive experiments on QVHighlights, TACoS and ActivityNet‑Caption demonstrate consistent performance gains under streaming settings.

Critical strengths
  1. This paper is well‑written and well‑organized. The problem background, pipeline, formulas, figures and appendices are clearly structured, making this complex streaming video grounding task easy to follow.
  2. The authors perform comprehensive evaluations over three benchmarks. They validate their plug‑in dual‑memory module on multiple backbones, alongside thorough ablations, noise‑robustness tests and failure‑case analysis, following solid empirical practice.
Critical weaknesses
  1. Limited technical novelty of the proposed dual‑memory design. The core idea of maintaining two levels of memory (short‑term within‑video history memory and long‑term cross‑video event memory) plus future‑trend prediction has been explored in prior streaming grounding work, specifically HEM [1].The intra‑memory is essentially an attention‑driven history compressor, while the header‑slider inter‑memory serves as a prototype bank for action patterns. Overall, the contributions appear incremental rather than conceptually groundbreaking.
  2. Missing comparison against modern multimodal large‑model baselines, weak justification for task necessity. Recent open multimodal models such as Qwen‑VL series support streaming video input and temporal event grounding. The paper does not include any benchmark against these strong real‑world baselines. Without such comparison, it becomes hard to justify why we need a dedicated specialized architecture for this streaming setting.

[1] Minghang Zheng, Yuxin Peng, Benyuan Sun, Yi Yang, Yang Liu. Hierarchical Event Memory for Accurate and Low‑Latency Online Video Temporal Grounding. ICCV, 2025

Questions for authors

Q1. Recent open multimodal large‑language models like Qwen‑VL support video temporal grounding. Could you explain why these MLLMs are absent from your baselines under the causal streaming evaluation protocol? Are there inherent barriers that stop MLLMs from being directly deployed for this streaming task? Please also discuss the pros and cons between your specialized architecture and MLLM‑based alternatives under streaming constraints.

Q2. The prior work HEM (Zheng et al., ICCV 2025) also leverages short‑term intra‑video memory and long‑term cross‑video event memory for streaming temporal grounding. Could you elaborate and quantify the key algorithmic differences between your dual‑memory framework and HEM? Please explicitly identify the genuinely novel components in your design, instead of incremental modifications upon HEM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.