acceptodds
Withdrawn from ICLR 2027

HMER: Hyperbolic Event Memory and Reasoning for Streaming Text-to-Video Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Existing text-to-video retrieval methods usually perform offline cross-modal matching on complete videos, which limits their applicability to streaming scenarios where videos arrive continuously and future content is unavailable. To address this issue, we study Streaming Text-to-Video Retrieval (STVR), which requires a model to identify relevant video streams while determining whether the target event is occurring at the current moment. This setting poses three main challenges: incomplete visual evidence, interference from past events, and unstable moment-level event predictions. To tackle these challenges, we propose Hyperbolic Memory-Based Event Reasoning (HMER), which jointly models event context, cross-modal hierarchy, and moment-level event states. Within HMER, we develop a Radial Valley-Guided Event Memory (RVG-EM) module to estimate event transitions from changes in hyperbolic radial trajectories. It aggregates historical information only within the current event, enriching local visual evidence while reducing cross-event interference. Since a video segment provides only local evidence for the complete event described by the text, we model the query and video segment as a general–specific semantic relation. Based on this view, we propose the Angular Cone Separation (ACS) loss to organize matching and non-matching representations with angular cone constraints, improving hierarchical alignment and cross-modal discrimination. Because semantic relevance alone cannot determine whether the target event is currently occurring, we further propose a Streaming Query-Aware Event Reasoning (SQ-AER) module. It jointly predicts query-matching confidence and event boundaries and combines them with historical high-confidence predictions for online event assessment. This enables the model to determine when retrieval results should be output. Experiments on DiDeMo, TVR, and Charades-STA show that HMER effectively models event context, cross-modal hierarchical relations, and moment-level event states, leading to consistent improvements on the STVR task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Text-to-Video Retrieval, Hyperbolic Representation Learning, Cross-Modal Retrieval, Streaming Video Understanding‌

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qKpEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Streaming Text-to-Video Retrieval (STVR) task, where a model continuously retrieves relevant videos from multiple synchronously arriving video streams and determines whether the queried event is occurring at the current moment. The authors propose HMER, which consists of three main components. RVG-EM uses radial valleys in hyperbolic space to estimate event transitions and aggregate historical segments within the current event. ACS introduces a query-centered angular cone to model the general-specific hierarchy between text queries and local video evidence. SQ-AER predicts query-matching confidence and event boundaries, and maintains the highest-confidence historical prediction to determine whether retrieval results should be triggered at the current moment.

Critical strengths

The paper formally introduces the STVR setting, which combines cross-stream retrieval with online event occurrence assessment in a multi-stream streaming scenario.

Critical weaknesses
  • I am not fully convinced by the necessity of hyperbolic space in this work. The paper uses radial valleys to estimate event transitions and angular cones to model the hierarchical relation between queries and videos, but the current motivation and evidence do not sufficiently explain why these problems should be modeled in hyperbolic space. In particular, the connection from radial semantic structure to radial valleys as event boundaries is not sufficiently clear, and the correspondence between cone geometry and semantic hierarchy in ACS also requires stronger justification.

  • The paper defines STVR around multiple synchronously arriving video streams, but the connection between this defining property and the proposed method is not sufficiently clear. RVG-EM mainly performs history aggregation within each individual stream, SQ-AER mainly performs event confidence prediction and boundary localization, and ACS does not appear to be specifically designed for the multi-stream streaming setting. Cross-stream retrieval itself is still mainly based on similarity ranking. It is therefore unclear why HMER is particularly required for the multi-stream streaming setting of STVR, and what its key methodological distinction is from existing streaming temporal grounding methods.

Questions for authors
  • Why is hyperbolic space necessary in this work? What is the specific motivation connecting hyperbolic geometry to the event transition modeling in RVG-EM?

  • How does the query-centered cone in ACS correspond to the semantic hierarchy?

  • Which components of HMER are specifically required by the multi-stream streaming setting of STVR? Why can existing streaming temporal grounding methods not be naturally adapted to this task?

  • What is the inference latency of the proposed framework under the streaming setting? In particular, since the event memory in RVG-EM grows from the most recently detected valley, how is its memory cost kept bounded in long-duration streaming scenarios when no event boundary is detected for an extended period?

Reviewer_oPCDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies streaming text-to-video retrieval, where a model must retrieve the relevant video stream and determine whether the queried event is currently occurring using only current and past observations. HMER partitions event memory using local valleys in hyperbolic radial trajectories, applies an Angular Cone Separation loss to query–video representations, and combines confidence and boundary predictions with retained historical predictions to control retrieval outputs. Experiments on DiDeMo, TVR, and Charades-STA report improvements over the evaluated baselines, supported by component ablations, memory comparisons, and geometric analyses.

Critical strengths
  1. Jointly identifying the relevant stream and determining whether the event is currently occurring, without future observations, addresses a meaningful application need.
  2. The complete method outperforms the reported baselines across all 18 evaluation settings on three datasets.
Critical weaknesses
  1. Positive cone inclusion and negative cone exclusion already appear in prior work [1], and HLFormer uses Lorentz attention and cross-modal cone constraints for video retrieval [2]. ACS changes the loss formulation and hierarchy direction, but lacks direct comparison with existing cone objectives. For RVG-EM, a relationship between radius and semantic generality does not establish that event transitions should produce radial valleys. Under the origin exponential map in Eq. (15), radial distance equals the norm of the input feature. The boundary coverage in Figure 5 and memory comparisons in Table 4 provide preliminary support, but without detection-count-matched controls or sliding-window and feature-change alternatives, they do not establish that the gains arise from the claimed event structure.
  2. The main comparisons use offline PRVR methods adapted through causal masking and threshold-based event prediction, without clarifying whether they receive equivalent per-step event and boundary supervision. In Table 3, adding SQ-AER alone raises DiDeMo R@100 at IoU=0.5 from 1.32% to 12.98%, indicating that online decision capability substantially affects performance. Existing online grounding methods already provide event memory, classification, and boundary regression [3], but neither their combination with retrieval nor representative PRVR baselines equipped with the same SQ-AER is evaluated.

[1] O.-E. Ganea, G. Bécigneul, and T. Hofmann, “Hyperbolic Entailment Cones for Learning Hierarchical Embeddings,” in Proceedings of the 35th International Conference on Machine Learning, PMLR, vol. 80, 2018, pp. 1646–1655. proceedings.mlr.press [2] J. Li, J. Wang, C. Tan, N. Lian, L. Chen, Y. Wang, M. Zhang, S.-T. Xia, and B. Chen, “Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 23074–23084. [3] M. Zheng, Y. Peng, B. Sun, Y. Yang, and Y. Liu, “Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 21589–21599.

Questions for authors
  1. What event and boundary supervision do the adapted baselines receive? Does HMER retain its advantage when representative PRVR methods use the same SQ-AER under matched features and supervision?
  2. With other components fixed, how does radial-valley memory compare with sliding-window and adjacent-feature-change strategies? Does the boundary coverage advantage persist when the number of detected boundaries is controlled?
  3. With other components fixed, how much does ACS improve over the standard positive-only entailment loss and an existing cone objective incorporating negative pairs?
Reviewer_F5MBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies streaming text-to-video retrieval (STVR): given a text query and multiple synchronously arriving video streams, the model must identify the relevant stream and decide whether the queried event is occurring now, using only observed content. HMER combines three components. RVG-EM detects local valleys in the hyperbolic radial trajectory and resets the memory window at those estimated event transitions. ACS places matching video representations inside a query-centered entailment cone and discourages nonmatching representations from entering it. SQ-AER predicts matching confidence and event boundaries, retains the highest-confidence historical prediction, and uses its interval to gate the current retrieval output. Training combines cone, cross-stream contrastive, confidence, and boundary-regression losses. Evaluation adapts DiDeMo, TVR, and Charades-STA to synchronized streaming and uses a joint retrieval-and-temporal-overlap metric. HMER improves over all reported adapted PRVR baselines.

Critical strengths
  1. The evaluation protocol jointly tests stream retrieval and online event occurrence: Section 4.1 (p. 8) penalizes false triggers outside the ground-truth interval as well as missed detections inside it while requiring the target stream to appear in the top-K results. Under this joint metric, HMER achieves large gains across all three datasets—at R@100 and IoU=0.5, it reaches 11.88 on TVR, 21.60 on DiDeMo, and 13.49 on Charades-STA versus 2.62, 1.30, and 0.22 for the best adapted PRVR baselines (Tables 1–2, pp. 8–9).

  2. Beyond the main comparison, Table 3 (p. 9) shows clear incremental gains from adding SQ-AER, RVG-EM, and ACS, and Appendix A.5 (Table 4, p. 16) isolates the memory and prediction-retention choices on DiDeMo: R@100 at IoU=0.7 improves from 7.36 with full-history memory to 11.94 with RVG-EM, and dropping historical maximum-confidence retention in favor of per-step prediction lowers it to 6.20.

Critical weaknesses
  1. RVG-EM maps features from the origin tangent space via exp_o(v) (Section 3.3; Appendix A.1, Eq. 15), the radial distance d(o, exp_o(v)) equals the Euclidean norm of the tangent vector, so valley detection essentially tracks a learned feature-norm curve. Figure 5 reports boundary coverage without controlling for how frequently memory resets occur, and Table 4 does not compare against a tuned sliding window or a simple feature-change/norm valley reset at a matched reset rate (even accounting for unannotated transitions noted in Appendix A.8). Likewise, while Figures 6–8 visualize norm distributions, HoroPCA projections, cone margins, and radial trajectories, they do not disentangle the claimed semantic hierarchy from modality norm offsets or the ACS loss; Please add a capacity-matched Euclidean baseline with the same losses would clarify what hyperbolic geometry actually contributes.

  2. Tables 1–2 cover DiDeMo, TVR, and Charades-STA, the source clips are short, shorter streams are artificially padded by cyclic repetition to match the longest stream, and evaluation stops at each target video's original duration (Appendix A.3). Moreover, while Section 2.2 and Section 3.5 directly motivate SQ-AER from online video grounding methods such as TSGSV [R2] and Zheng et al. [R3], the experiments only compare against offline PRVR models; Could you adapt [R2, R3] to the multi-stream setting under shared features and the same joint metric, and testing or discussing continuous unpadded streams such as Ego4D? This would better reflect realistic online streaming.

  3. Please report training and per-step inference FLOPs, wall-clock step latency, and peak GPU memory relative to the adapted baselines as stream count N_v and candidate size K scale to better prove the effectiveness and efficiency of your proposed method.

References:

[R1] Grauman et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022. https://arxiv.org/abs/2110.07058

[R2] Gan et al. Temporal Sentence Grounding in Streaming Videos. ACM MM 2023. https://arxiv.org/abs/2308.07102

[R3] Zheng et al. Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding. ICCV 2025. https://arxiv.org/abs/2508.04546

Questions for authors
  1. How do the strongest PRVR baselines (e.g., JSG or GMMFormer) perform when equipped with the exact same SQ-AER confidence/boundary heads, losses, and historical retention rule as HMER? Also, how were the event-occurrence thresholds chosen for the baselines in Tables 1–2?

  2. What is the average reset frequency of RVG-EM across threshold values tau in Figure 4–5, and how does HMER compare against a sliding window of matched average length or a Euclidean norm-valley reset under the same training objective?

  3. In Appendix A.2, the retained historical prediction is updated only when a newer step achieves higher confidence; if an early step produces a false-positive high confidence score, how does the model recover once that predicted interval passes? Please also specify the segment step size, top-K candidate size |C_t|, and total concurrent stream count N_v used at test time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.