acceptodds
Under review as a conference paper at ICLR 2027

Event-ReHead: Event-Aware Head Retrieval and Memory-Augmented Streaming Video Understanding

Abstract

Vision Language Models (VLMs) have achieved strong performance in video understanding, while streaming video question answering remains challenging due to the need for accurate retrieval under strict memory and latency constraints. Existing KV-cache-based methods mainly organize video memory as independent visual tokens, which ignores underlying temporal semantics of videos and leads to fragmented retrieval, failing to capture continuous evidence . We propose Event-ReHead, a training-free framework that improves streaming video understanding by introducing event semantics into KV-cache retrieval. Our method preserves the efficiency of frame-level retrieval while reorganizing retrieved evidence into an *event skeleton*, a compact and coherent representation capturing the underlying temporal structure. Building on this, we introduce a query-specific attention head selection strategy to construct more discriminative retrieval representations, enabling precise alignment with question-relevant visual signals. We further incorporate a lightweight interaction memory that selectively reuses prior question-answer context, providing useful temporal priors without introducing noise. We evaluate Event-ReHead on four streaming and three offline benchmarks, demonstrating consistent improvements across diverse VLM backbones. In streaming settings, it achieves up to +4.1% gains, while also yielding consistent improvements in offline scenarios, with an average gain of around +2-3%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Streaming Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hT6ZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Event-ReHead, a training-free framework for streaming video understanding. It combines query-specific attention head selection, question-conditioned event segmentation, event skeleton augmentation, and selective reuse of historical question–answer pairs. This method utilizes representative frames from relevant events to complement token-level KV cache retrieval, aiming to enhance temporal consistency within a limited memory budget. Experiments cover multiple streaming and offline video benchmarks and employ various VLM backbones.

Critical strengths
  1. The motivation is reasonable: retrieving only visual tokens related to specific content risks losing the temporal context necessary to understand events; incorporating event-level context to complement retrieval effectively addresses a practical limitation in streaming video QA.
  2. The framework operates without updating backbone network parameters and reuses existing visual representations, offering potential value for adaptation to VLMs.
Critical weaknesses

1. There are inconsistencies between the main text and the tables regarding several key results, complicating the assessment of the actual performance gains. In Table 1, the OVO-Backward score for LLaVA-OV-0.5B is listed as 43.53; however, the corresponding score in Table 5 is 35.24, with 43.53 appearing in the "Overall" column instead. For Qwen2.5-VL-7B, Table 5 reports an "Overall" score of 61.92, yet the scores for its two branches are 72.51 and 50.19; based on the averaging method used in other rows of the table, the result should be 61.35. Furthermore, several improvement figures cited in Section 4.1 cannot be recalculated using the data in Table 1. These discrepancies materially affect the paper's main conclusions. Moreover, the ablation results in Figure 4 are inconsistent with the main tables. In the OVO-Bench panel, judging by bar heights, the result for the full model labeled LLaVA-OV-7B is over than 70, while the result for Qwen2.5-VL-7B is around 66; yet, the corresponding real-time perception scores in Table 1 are 67.02 and 72.51, respectively. These bar values ​​also fail to align with the "Overall" scores in Table 5. Additionally, the claim of "consistent improvement" requires qualification: Table 1 shows that Qwen2.5-VL-32B experienced a 2.33 percentage point drop on VideoMME compared to HERMES.

2. Key implementation details are incomplete, making it difficult to confirm how the method adheres to the fixed cache budget. With a 4000-token cache budget, the KV data for certain event skeleton frames may have already been evicted; however, the paper does not specify how this KV data is retrieved—whether it is stored elsewhere, requires re-computation, or must be selected from the currently retained cache. Equation (14) incorporates event skeleton KV data via set union but fails to define how the total budget constraint is enforced after this addition, nor does it clarify the specific function of the "event ratio = 0.4" parameter. Further explanation is also required regarding the construction of frame representations and question representations in Equation (6). These details directly impact the method's reproducibility and resource overhead.

3. Efficiency experiments are insufficient to validate the actual overhead of the newly introduced components. In Figure 3, the TTFT y-axis is labeled in milliseconds with values ​​below 4, yet the data points for the proposed method fall in the 27–29 ms range; this discrepancy between units and plotted values ​​requires correction. More importantly, the efficiency comparison omits HERMES, the most direct baseline. The paper also fails to clarify whether the TTFT calculation accounts for operations such as attention head selection, question-conditioned event discovery, event skeleton retrieval, and historical QA retrieval. Merely reporting stable GPU memory usage is insufficient to demonstrate the total cost associated with additional historical frame features or KV storage.

Questions for authors
  1. Specify which set of results in Tables 1 and 5 is the definitive one. Explain the reasons for the score inconsistencies, provide the aggregation rules used for the "Overall" metric, and ensure that the improvement figures cited in Section 4.1 align with the corrected tables.
  2. How are the KV pairs for event skeleton frames retrieved after cache eviction? Explain the storage or recovery mechanism, the budget execution method following Equation (14), the specific meaning of the "event ratio," and the construction methods for the visual and textual representations in Equation (6).
  3. Specify the correct units for Figure 3 and the timing range for TTFT. Which operations dependent on the current question are included in the latency calculation? Are there existing measurement results directly comparable to HERMES under the same hardware, backbone, and cache budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hT6Z (Part 1/3)

Response to Question 1: Numerical consistency and ablation corrections

Thank you for carefully checking the text, tables, and figures. We acknowledge that these inconsistencies undermine the verifiability of the results. We will correct the numerical entries, replace the erroneous ablation plot, and revise the associated claims.

Table corrections. For LLaVA-OV-0.5B on OVO-Bench Backward, the entry in Table 1 should be 35.24, not 43.53; the latter was copied from the Overall column of Table 5. For Qwen2.5-VL-7B on OVO-Bench, the Overall score in Table 5 should be 61.35, not 61.92.

We will recalculate the gains in Section 4.1 and the appendix and check the corresponding statements in the abstract, introduction, and conclusion.

Figure 4 correction. In the OVO-Bench panel, the group labeled LLaVA-OV-7B incorrectly used Qwen2.5-VL-7B results on OVO-Bench Real-Time. The group labeled Qwen2.5-VL-7B incorrectly used MVBench ablation data. The intended benchmarks were StreamingBench, OVO-Bench Real-Time, and VideoMME. We will correct the panel, identify the OVO-Bench split explicitly, and revise Section 4.3.

We supplement the corrected ablations using LLaVA-OV-7B and Qwen2.5-VL-7B on OVO-Bench Real-Time. Each variant removes one component while retaining the others; all entries are answer accuracy (%):

Backbonew/o Memoryw/o Eventw/o HeadEvent-ReHead
LLaVA-OV-7B66.3864.7365.5767.02
Qwen2.5-VL-7B71.9270.7971.7872.51

Removing memory, event retrieval, and head selection reduces accuracy by 0.64, 2.29, and 1.45 percentage points for LLaVA-OV-7B, and 0.59, 1.72, and 0.73 points for Qwen2.5-VL-7B. All three components contribute positively in this split, with the largest reduction following removal of the event module.

Separate MVBench results. We will report the offline ablations separately in the appendix. We add ablations using LLaVA-OV-7B and Qwen2.5-VL-7B on MVBench to examine whether the components also benefit offline video understanding. Entries are answer accuracy (%):

Backbonew/o Memoryw/o Eventw/o HeadEvent-ReHead
LLaVA-OV-7B56.9857.3457.1858.79
Qwen2.5-VL-7B66.0165.4365.3766.65

The full model exceeds each ablation in this table, but the relative contributions vary across backbones and tasks. Through ablation experiments on MVBench, we further demonstrate the roles of our various components.

VideoMME exception. We agree that the claim of consistent improvement was too broad. With Qwen2.5-VL-32B on VideoMME, Event-ReHead scores 63.11, compared with 65.44 for HERMES, a decrease of 2.33 points.

One possible explanation is a trade-off between selective event retrieval and global coverage. Event-ReHead concentrates the cache on question-relevant evidence and supplements temporal continuity within a relevant event. Some offline questions, however, require evidence distributed across multiple events. More concentrated retrieval may reduce coverage elsewhere.

The limitation is specific to the evaluated backbone–dataset combination. On VideoMME, Event-ReHead improves over HERMES by 0.56 points with LLaVA-OV-7B and 1.31 points with Qwen2.5-VL-7B. With Qwen2.5-VL-32B, it improves StreamingBench, OVO-Bench Real-Time, and OVO-Bench Backward by 0.57, 0.50, and 1.14 points, and MVBench and EgoSchema by 1.43 and 3.61 points.

We will therefore state that Event-ReHead improves the reported streaming settings and most offline settings, rather than every backbone–dataset combination. We take responsibility for the presentation errors and will cross-check tables, plots, and textual gains against the original evaluation outputs.

AuthorsOpenReview ↗

Response to Reviewer hT6Z (Part 2/3)

Response to Question 2: Event skeletons, retrieval budget, and representations

Skeleton construction. For the selected event Ej∗E_{j*}, we divide its duration into three consecutive stages: beginning, middle, and end. Within each stage, we select the frame with the highest question–frame relevance from Eq. (6):

tk∗=arg⁡max⁡t∈Ej∗kρt(q)t_k^*=\arg\max_{t\in E_{j*}^k}\rho_t(q),

where k∈{1,2,3}k\in\{1,2,3\} indexes the three stages. The resulting skeleton is Sj∗={t1∗,t2∗,t3∗}S_{j^*}=\{t_1^*,t_2^*,t_3^*\}.

The temporal stages determine where to search; question relevance determines which frame represents each stage. This preserves coverage of the event progression without mechanically selecting boundary frames. Candidates must have available KV evidence.

Event ratio. In the clarified budget-allocation specification, γ\gamma limits the share of the retrieval budget assigned to event evidence: Be=⌊γBr⌋B_e=\lfloor\gamma B_r\rfloor, where BrB_r is the total visual-token retrieval budget and BeB_e is the maximum number of positions allocated to event evidence.

Thus, γ=0.4\gamma=0.4 allows event evidence to occupy at most 40% of that budget. It neither discards 60% of the skeleton anchors nor permanently reserves 40% of the cache. All three stages participate; the amount of evidence depends on coverage by the base retrieval.

Completion within the budget. Skeleton evidence comes only from KV positions still available in the current cache. We neither recover evicted historical KV nor re-encode the corresponding frames. We first allocate representative evidence to uncovered event stages, then fill the remaining budget with high-scoring base evidence. Overlapping positions are counted only once, and the final set satisfies ∣Rℓ∣≤Br|R_\ell|\le B_r.

For a 4,000-token budget, event evidence can occupy at most 1,600 positions. If only 300 positions are needed, the remaining budget remains available for fine-grained retrieval. The objective is to improve temporal coverage within the same budget, rather than retain an additional historical KV store.

Frame and question representations. For a frame or temporal bin tt, we average the projected visual tokens associated with that frame:

ft=1∣It∣∑i∈Itzif_t=\frac{1}{|I_t|}\sum_{i\in I_t}z_i.

The visual features ziz_i are produced by the backbone vision-to-language projection and are retained as compact frame-level features for event analysis. They are not additional per-layer KV pairs.

For a question qq, we tokenize the text, obtain the corresponding input embeddings, and average them:

uq=1M∑m=1ME(qm)u_q=\frac{1}{M}\sum_{m=1}^{M}E(q_m).

After L2 normalization, the frame-question relevance is computed as:

ρt(q)=ft⊤uq∥ft∥2∥uq∥2\rho_t(q)=\frac{f_t^\top u_q}{\|f_t\|_2\|u_q\|_2}.

This similarity is used as a lightweight semantic cue together with visual frame-change scores during question-conditioned event discovery and event selection. It does not require an additional vision-encoder pass. The final KV selection remains based on the contextualized retrieval scores used by the cache-retrieval module. This relevance guides query-conditioned event discovery, allowing the model to locate the relevant event before performing KV retrieval and to reduce temporal fragmentation caused by concentrating retrieval on only a few instants.

AuthorsOpenReview ↗

Response to Reviewer hT6Z (Part 3/3)

Response to Question 3: Efficiency measurements and timing scope

Figure 3. We will correct the TTFT axis and annotations. If the axis is displayed in seconds, values of 27–29 ms will be plotted as 0.027–0.029 s, with any millisecond annotations explicitly labeled.

HERMES comparison. We supplement the efficiency evaluation using Qwen2.5-VL-7B, with 8 frames per chunk and input lengths of 16, 64, 256, and 512 frames. Both methods use the same input sequences, KV budget, hardware, and decoding configuration. This controlled workload comparison measures GPU memory, first-token latency, and subsequent decoding latency; the frame counts describe the workload rather than an accuracy benchmark.

FramesMethodGPU memory (GB)TTFT (ms)TPOT (ms/token)
16HERMES16.5427.0124.43
16Event-ReHead16.8027.5224.56
64HERMES16.6628.4123.89
64Event-ReHead16.8328.7124.31
256HERMES16.6628.4424.02
256Event-ReHead16.8328.7124.45
512HERMES16.6628.4123.98
512Event-ReHead16.8328.7124.51

At 512 frames, Event-ReHead adds 0.17 GB memory, 0.30 ms TTFT, and 0.53 ms TPOT, approximately 1.0%, 1.1%, and 2.2%, respectively. Our claim is that retrieval quality improves at a modest measured cost, not that Event-ReHead is more efficient than HERMES.

Timing scope. We measure TTFT from the moment the user query is received, before any query-dependent processing begins, until the first answer token is produced. The measured interval includes attention-head selection, question-conditioned event discovery, event-skeleton mapping and retrieval, historical QA retrieval, cache selection, prompt construction, answer prefill, and the computation required to generate the first answer token. Only video processing completed before the query arrives is excluded. This end-to-end query-level timing allows the reported TTFT to capture the additional processing introduced by Event-ReHead when compared with HERMES.

Storage scope. The GPU KV cache stays within a fixed budget, so GPU memory remains stable as the video grows. We also keep one small feature vector per processed frame or temporal segment for event detection. This auxiliary storage grows with video length. Thus, our results show a small GPU-memory and latency overhead in the tested setting.

Declaration of Contribution

We are the first to supplement missing temporal information in streaming videos under a fixed budget by completing event skeletons. Combined with attention-head retrieval and an interaction memory bank, our method improves the performance of base models on mainstream benchmarks with low additional cost. We believe this approach has practical value and hope the reviewers will recognize its contributions and reconsider the paper.

Reviewer_Z6vAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a training-free framework designed to improve streaming video understanding in VLMs under memory and latency constraints. It proposes a novel solution that combines fine-grained attention mechanisms with video structural priors at inference time. It uses query specific attention head selection, question conditioned event discovery & retrieval, and lightweight interaction memory to reduce semantic fragmentation and improve continuous event evidence. Evaluation across multiple benchmarks and using different backbones shows the proposed method achieves up to 4.1% performance gains.

Critical strengths
  1. The paper proposes a training-free framework designed to improve streaming video understanding in VLMs under memory and latency constraints.
  2. It introduces a novel solution which reduces semantic fragmentation via structure aware evident retrieval.
  3. The proposed solution maintains stable GPU memory usage as video duration increases, making it well-suited for the targeted applications.
  4. The paper presented comprehensive evaluation results.
Critical weaknesses
  1. The paper included GPU memory and TTFT analysis. I would encourage the authors to add end-to-end latency and cache maintenance analysis as well to strengthen the claim.
  2. Is there a theoretical foundation to the head selection and event granularity decision?
  3. The paper seems to have used fixed setting for evaluation. Is this sufficient for the performance analysis, especially since the paper reported that optimal choices may vary by benchmark, video length?
  4. Is the top-K attention head selection strategy an optimal choice? Does it have any correlation with downstream task performance?
Questions for authors
  1. Is there any theoretical foundation to the head selection and event granularity decision?
  2. Is the top-K attention head selection strategy an optimal choice? Does it have any correlation with downstream task performance?
  3. The interaction memory stores previous question-answer pairs and cosine similarity is used to compare the new query with historical ones. Does this strategy introduce incorrect answers from earlier ones?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Z6vA (Part 1/2)

Response to Question 1: End-to-end latency and cache maintenance

We add an efficiency comparison between HERMES and Event-ReHead using Qwen2.5-VL-7B. Both methods process the same video inputs with identical KV-cache budgets, hardware, and decoding settings. We use 8 frames per chunk and vary the total input length from 16 to 512 frames to examine how resource usage changes as the observed video history grows.

The table reports GPU memory usage, time to first token (TTFT), and time per output token (TPOT). TTFT covers the interval from the start of question-dependent processing to the first answer token, including head selection, event discovery, skeleton retrieval, cache selection and compression, historical QA retrieval, and answer prefill. TPOT measures the average latency of subsequent output tokens.

FramesMethodGPU memory (GB)TTFT (ms)TPOT (ms/token)
16HERMES16.5427.0124.43
16Event-ReHead16.8027.5224.56
64HERMES16.6628.4123.89
64Event-ReHead16.8328.7124.31
256HERMES16.6628.4424.02
256Event-ReHead16.8328.7124.45
512HERMES16.6628.4123.98
512Event-ReHead16.8328.7124.51

Event-ReHead’s GPU memory and latency remain nearly stable over the tested input lengths. At 512 frames, it adds 0.17 GB of GPU memory, 0.30 ms of TTFT, and 0.53 ms of TPOT relative to HERMES—approximately 1.0%, 1.1%, and 2.2%, respectively. These results show that the complete method introduces modest measured overhead while preserving low first-token and decoding latency. The table measures the aggregate cost of Event-ReHead, rather than attributing the difference to any single component.

To provide a more comprehensive evaluation of response overhead, we additionally compare the TTFT, decoding time, and total response time of HERMES and Event-ReHead on Qwen2.5-VL-7B. Both methods use the same input (512 frames), cache configuration, and decoding protocol, with the output length fixed at 64 tokens to eliminate variations caused by response length.

MethodTTFT (ms)TPOT (ms/token)Decoding Time for Remaining 63 Tokens (ms)Total Response Time (ms)
HERMES28.4123.981510.741539.15
Event-ReHead28.7124.511544.131572.84
Increase+0.30+0.53+33.39+33.69 (~2.19%)

Under this setting, Event-ReHead increases the total response time by approximately 2.2% compared with HERMES, indicating that the additional mechanisms introduce only a small overall latency overhead. This is because event discovery reuses existing pooled frame features, while skeleton retrieval operates through indices into the current KV cache, without re-encoding historical video frames. Subsequent generation follows the backbone model's original autoregressive decoding process. Therefore, our method primarily changes how historical evidence is selected and organized, without introducing an additional full-video encoding stage or answer-generation stage.

Response to Question 2: Rationale for head selection and event granularity

Our design is motivated by attention-head heterogeneity and the temporal structure of continuous activity. We do not claim a theorem guaranteeing optimal downstream QA accuracy.

Head selection. Prior work on attention-head specialization and pruning, including Voita et al. (ACL 2019) [1], Michel et al. (NeurIPS 2019) [2], Retrieval Heads [3], and DuoAttention [4], motivates treating heads differently rather than aggregating them uniformly. Event-ReHead scores each head using its question-to-visual attention response and selects Top-KK heads to emphasize relevant visual signals.

Event granularity. Work on event perception and video boundary detection, including Zacks et al. (2007) [5], Shou et al. (ICCV 2021) [6], and Tan et al. (TPAMI 2023) [7], motivates organizing continuous activity into temporally coherent units. Event-ReHead combines visual and semantic changes to identify boundaries and supplements fine-grained retrieval with event structure.

These studies provide conceptual and empirical motivation, rather than a proof that our particular scoring rule or event granularity is universally optimal. We will clarify this distinction and provide complete references.

[1] Voita et al., ACL 2019, Analyzing Multi-Head Self-Attention.

[2] Michel et al., NeurIPS 2019, Are Sixteen Heads Really Better than One?

[3] Wu et al., ICLR 2025, Retrieval Head Mechanistically Explains Long-Context Factuality.

[4] Xiao et al., ICLR 2025, DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads.

[5] Zacks et al., 2007, Event Perception: A Mind/Brain Perspective

[6] Shou et al., ICCV 2021, Generic Event Boundary Detection: A Benchmark for Event Segmentation

[7] Tan et al., TPAMI 2023, Temporal Perceiver: A General Architecture for Arbitrary Boundary Detect.

AuthorsOpenReview ↗

Response to Reviewer Z6VA (Part 2/2)

Response to Question 3: Top-K selection and downstream behavior

We assess its contribution through task-specific ablations and the consistency of selected heads.

Task-specific ablations. Using LLaVA-OV-7B on StreamingBench-RT, we group questions into Perception, Dynamic, Event, and Causal/Predictive families:

VariantPerceptionDynamicEventCausal/PredictiveOverall
w/o Head73.5356.3779.4077.8374.13
w/o Event74.0855.2376.3474.9572.97
Full Event-ReHead74.4356.7179.6078.0574.70

Head selection provides its largest category-level gain on Perception (+0.90 points), consistent with its role in retrieving question-relevant objects, attributes, text, and spatial details. The event module contributes most to Event (+3.30) and Causal/Predictive (+3.10) questions, supporting its role in temporal evidence organization.

Selection consistency. We add an analysis on StreamingBench and OVO-Bench using Qwen-2.5vl-7b to examine whether selected heads vary systematically with question category. We measure the Jaccard similarity between Top-8 attention heads selected for question pairs from the same category and from different categories. Random size-matched heads provide a reference baseline for evaluating whether the selected heads exhibit category-specific consistency.

DatasetRandom-K referenceCross-category pairsWithin-category pairs
StreamingBench0.160.190.35
OVO-Bench0.160.200.36

Similar questions tend to select more similar attention heads. This demonstrates that the selection mechanism for our attention heads is correlated with the category of the query, which facilitates more accurate retrieval in downstream tasks by utilizing more relevant attention-head representations.

Response to Question 4: Shared settings and sensitivity to video duration

We use shared settings to test transfer without benchmark-specific tuning, not to claim that one configuration is optimal everywhere. The paper analyzes cache budget, selected-head count, event ratio, and event count across datasets. We will distinguish the transferability demonstrated by a shared configuration from the task-specific preferences revealed by sensitivity analysis.

To examine video-length sensitivity, we specify an additional comparison using Qwen2.5-VL-7B on VideoMME, with the KV budget, FPS, and other parameters held fixed.

Target event countShort (%)Medium (%)Long (%)Equal-weight mean (%)
273.262.652.162.6
4 (shared setting)72.963.253.863.3
872.1062.954.463.1

The experiments results demonstrate that shorter videos may favor coarser segmentation, whereas longer videos may benefit from finer granularity. In the table, four target events perform best on Medium and remain within 0.30 and 0.60 percentage points of the best settings on Short and Long, respectively, while attaining the highest equal-weight mean.

A shared setting can provide a balanced compromise without being optimal for every duration group. The main evaluation retains a shared configuration; duration-specific comparisons characterize sensitivity rather than replace the main results with separately optimized settings. We will distinguish practical robustness over the tested range from universal optimality.

Response to Question 5: Error propagation from interaction memory

The contribution of the interaction memory is supported by ablations on multiple benchmarks. In the original StreamingBench ablation, removing the memory bank reduces the accuracy by 1.20 points for LLaVA-OV-7B and 1.18 points for Qwen2.5-VL-7B. Our corrected OVO-Bench Real-Time ablation shows additional decreases of 0.64 and 0.59 points, respectively. We also add a separate MVBench ablation, where removing memory decreases the accuracy by 1.81 points for LLaVA-OV-7B and 0.64 points for Qwen2.5-VL-7B.

The consistent performance drop across streaming and offline benchmarks indicates that the retrieved historical interactions generally provide useful complementary context for reasoning about the video and its events. The size of the gain varies across backbones and tasks, but the full model outperforms the memory-free variant in all reported settings. These experiments evaluate normally recorded interaction histories; robustness to deliberately incorrect historical answers remains an important direction for future work.

Thank you for the additional experiments. I have no additional comments or questions.

Dear Reviewer Z6vA

We sincerely appreciate your positive feedback. We are pleased that our response has addressed your concerns, and we are grateful for your recognition and support of our work.

Best regards,

Authors

Reviewer_ivquOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets streaming video question answering, where a model must encode frames incrementally, hold a bounded memory, and retrieve evidence when a query arrives at an arbitrary time. Its premise is that existing KV-cache retrieval methods treat the cache as a flat collection of visual tokens or blocks, which the authors argue causes three problems: fragmented evidence that ignores the event structure of video, dilution of query-relevant signal when retrieval keys are built by aggregating all attention heads uniformly, and no reuse of earlier interactions. Event-ReHead is a training-free framework addressing all three on top of a HERMES-style compact cache. First, it scores each attention head by its peak attention from query tokens to cached visual tokens, keeps the top-K heads, and computes token relevance over only those heads before blending with HERMES's inherited recency prior. Second, it discovers query-conditioned event boundaries by summing a visual transition score (dissimilarity between adjacent frame features) and a semantic transition score (the change in query relevance across a candidate boundary), selects the single event with the highest density of retrieved frames, and augments the base retrieval with a three-frame "event skeleton" drawn from that event's beginning, middle, and end — added only when those frames are not already covered by the base retrieval. Third, a small bank of prior question–answer pairs is matched against the incoming query by sparse token-vector similarity and prepended as short context. Across four streaming and three offline benchmarks and four backbones, the method reports consistent gains over HERMES (averages of +1.50 to +2.11 points), with GPU memory that stays flat as frame count grows and time-to-first-token comparable to ReKV. The direction is timely and the empirical coverage is broad, but the central mechanisms are specified too loosely to interpret — what a detected boundary actually corresponds to, and why a fixed three-anchor skeleton should suffice for events of arbitrary length — and the efficiency analysis does not isolate the cost of the proposed retrieval from the cache design it inherits, nor compare the memory organization against the closest prior work on equal terms.

Critical strengths

Strengths

The paper is well written and well motivated, the direction of the paper is good and addresses the timely problem of retrieval under strict memory and latency constraints.

The results are also promising, specially the results in table 1 show pretty consistent gains.

The analysis on efficiency indicates that the model is efficient in both ram usage and memory usage.

Critical weaknesses

Weakness

In section 3.2 does the boundary change imply change in motion? Analytically what is being selected? 

In eqn 12 , why is the beginning, middle and ending stages of the event selected? If the event is long vs small, wouldn't that mean missing important information?

How much additional compute is required to retrieve from memory? Also how does the design of the memory compare to related works to ReKV Streaming video question-answering with in-context video KV-cache retrieval. In ICLR, 2025),  HERMES: KV cache as hierarchical memory for efficient streaming video understanding. In ACL, 2026. Comparison to the other memory based method and comparing the design of memory choice will be important for the paper.

As the FPS is increasing the results are actually getting smaller, for ex. In table 1 in o.5 FPS the results are stronger vs in 1 FPS results are weaker.

Adding compact reusable cache with event-level memory, take how much additional computational cost?

Questions for authors
  1. There are some issues in Table 1. For LLaVA-OV-0.5B , the OVO-Bench-Backward column reads 43.53, but Table 5 shows that row's Backward average is 35.24 and its Overall average is 43.53. The main table appears to report the overall score in the backward column, which turns a +0.44 improvement over HERMES into an apparent +8.73.
  2. Efficiency comparison is only with ReKV , not with HERMES.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ivqu (Part 1/2)

Response to Question 1: Visual changes versus semantic event boundaries

The score in Section 3.2 is not optical flow or motion magnitude. Visual changes can reflect actions, scene transitions, or viewpoint changes; not every change is treated as a semantic boundary. We combine visual transitions with changes in question relevance to obtain query-conditioned event boundaries.

We add a direct localization evaluation on the ActivityNet-derived subset of Streamo-Instruct-465K, comprising 3,739 videos and 5,038 event queries. We compare the selected temporal interval with the annotated interval. R@1 measures localization success at IoU ≥ 0.5, mIoU measures interval overlap, mean boundary error measures temporal deviation, and F1@5s evaluates boundary detection within a five-second tolerance:

MethodR@1, IoU ≥ 0.5 (%) ↑mIoU (%) ↑Mean boundary error (s) ↓F1@5s (%) ↑
Uniform segmentation16.721.919.521.4
Visual-only boundaries27.829.814.334.8
Semantic-only boundaries33.633.711.841.7
Joint boundaries38.936.99.949.6
Full Event-ReHead42.439.58.653.2

Combining visual and semantic cues improves mIoU from 29.8 to 36.9 and reduces boundary error from 14.3 to 9.9 s relative to visual-only detection. This supports the semantic cue's usefulness beyond generic visual change. The full pipeline additionally includes event selection, so its further gain should not be attributed solely to boundary detection.

Response to Question 2: Sparse skeletons and long events

Beginning, middle, and end anchors provide temporal coverage; they are not assumed to contain every important detail. The base retrieval set RlbaseR_l^{base} remains the main source of fine-grained evidence. The event skeleton provides sequential signals for these fine-grained, frame-level features, thereby effectively signaling the continuous events within a video.

We add a controlled skeleton-strategy comparison using Qwen2.5-VL-7B on StreamingBench and VideoMME. No skeleton disables completion; Random-3 uses three random frames; Query-Top3 uses the three most question-relevant frames without explicit stage coverage; Event-ReHead combines fine-grained retrieval with event-skeleton completion. Accuracy and incremental TTFT relative to no skeleton are reported below:

Skeleton strategyStreamingBench (%)VideoMME (%)Extra TTFT vs. no skeleton
No skeleton78.9262.610 ms
Random-378.0862.70+0.02 ms
Query-Top379.3162.78+0.07 ms
Event-ReHead79.9663.31+0.11 ms

Based on the supplementary experimental results above, we can see that, compared to the baseline without event structure, our event completion strategy can achieve superior performance on both the StreamingBench and long video bench VideoMME by utilizing event skeletons. This indicates that our supplementary event frames can effectively enhance the model's performance. We acknowledge that a fixed-size skeleton can miss internal structure in long, complex events and will state this limitation explicitly.

Response to Question 3: Retrieval costs and memory-design differences

  • ReKV retrieves question-relevant frame/block KV from historical video memory. Preserving a more complete history can retain fine details, but incurs historical storage and data-movement costs, including CPU offloading in the compared setting.

  • HERMES continuously compresses historical KV through hierarchical relevance–recency management. It emphasizes reusable, compact memory, but token-level compression can leave temporally fragmented evidence and cannot directly recover evicted details.

  • Event-ReHead explicitly organizes the retained evidence along the event timeline. It selects question-relevant attention heads, routes retrieved tokens to a question-conditioned event, and supplements currently available skeleton evidence when coverage is insufficient. Pooled frame features and compact event information support this organization; they are not a second store of evicted per-layer KV. A bounded QA memory separately supplies related interaction history.

Thus, ReKV emphasizes retrieval from historical KV, HERMES emphasizes hierarchical compression, and Event-ReHead emphasizes question relevance together with event-level temporal coverage. The aggregate efficiency comparison is provided in our response to Question 5 below.

AuthorsOpenReview ↗

Response to Reviewer ivqu (Part 2/2)

Response to Question 4: FPS settings across backbones

  • Overall, Qwen (1 fps) performs better than LLaVA (0.5 fps), with only a limited number of metrics showing the 1 fps model underperforming the 0.5 fps model. A similar situation has also occurred with some of the data in HERMES.

  • The 0.5-FPS and 1-FPS rows in Table 1 correspond to different backbone settings, not a controlled experiment that changes only FPS. They therefore do not establish that increasing FPS reduces accuracy. We use 0.5 FPS for LLaVA-OV and 1 FPS for Qwen-2.5-VL, following the corresponding HERMES/ReKV evaluation protocols. We will identify the exact protocol sources throughout the paper and appendix.

  • Backbones differ in visual encoding, spatial/temporal token organization, and multimodal projection. The same frame count can therefore produce different token counts and cache pressure. Accuracy also depends on the base model and retrieval strategy. Our primary comparisons hold the backbone and sampling protocol fixed; cross-backbone comparisons do not isolate the effect of FPS. Higher FPS is not guaranteed to improve accuracy under a fixed budget: it may capture brief details but also increases redundancy and compression pressure. We will elaborate on this phenomenon in our appendix.

Response to Question 5: Additional computational and storage costs

We supplement the efficiency comparison using Qwen2.5-VL-7B on matched video-input workloads of 16–512 frames, with 8 frames per chunk. Both methods use the same input sequences, KV budget, hardware, and decoding settings. GPU memory measures device storage, TTFT measures query-to-first-token latency, and TPOT measures average subsequent token latency.

FramesMethodGPU memory (GB)TTFT (ms)TPOT (ms/token)
16HERMES16.5427.0124.43
16Event-ReHead16.8027.5224.56
64HERMES16.6628.4123.89
64Event-ReHead16.8328.7124.31
256HERMES16.6628.4424.02
256Event-ReHead16.8328.7124.45
512HERMES16.6628.4123.98
512Event-ReHead16.8328.7124.51

At 512 frames, Event-ReHead adds 0.17 GB GPU memory, 0.30 ms TTFT, and 0.53 ms TPOT, approximately 1.0%, 1.1%, and 2.2%, respectively. These figures measure the full method's incremental cost, not the event module in isolation. From the experiments above, we can see that our method does not cause significant time consumption.

The GPU KV cache stays within a fixed budget, so GPU memory remains stable as the video grows. We also keep one small feature vector per processed frame or temporal segment for event detection. This auxiliary storage grows with video length, but it is much smaller than the full KV cache and cannot recover evicted KV pairs. Thus, our results show a small GPU-memory and latency overhead in the tested setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.