- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops a method, FlashBack, to deal with the utilization of historical information in streaming video QA. It tries to enhance the ability of VLMs to make use of such history via a deliberately designed mechanism - to activate the route of history recall vs. current state. The authors reported positive outcomes of FlashBack using two VLM backbones (StreamingVLM, Mega-VL) on selected tasks of two benchmarks (SteamingBench and OvoBench).
Critical strengths
The paper is well-motivated on using history to answer visual questions in long videos. A training-free mechanism with frozen-VLM is notable and valid. The overall process is clearly explained and plausible. The design of experiment is in general valid by including multiple (2) VLMs and on benchmarks (2). The ablation study is also relevant to attribute the effect to the components, so as to explain the underlying mechanism.
Critical weaknesses
- The methodology – FlashBack- while well-motivated and clearly presented, could be brittle by adopting a layered/sequential process that separates the inferencing into temporal evidence routing and recall (answer generation). Specifically, for temporal evidence routing, Eq. 5 proposes a few categories of context space and their dispatching to different semantic-action mapping. First, there is no justification of the basis of these context categories and whether they are mutually exclusive and self-containing. There is neither reported nor empirical evidence on the design of these categories. Second, during inference stage, if the classification is wrong, the error will propagate thus voiding subsequent recall. This can be partially shown by the small margin (sometime mixed results) of the proposed approach overall. Last, the mechanism of “know when to remember”, which is a central component of the methodology, shows limited technical merit by using a naïve process of prompting and aggregating the response. Therefore, the novelty and significance of the work is limited.
- The main results show limited advantage of the proposed approach. While I appreciated the author’s honesty in reporting the results, the margin of FlashBack is very small and sometimes inferior to the benchmarks. On OVO-Bench, compared with the non-FlashBack baselines, only CRR exhibited substantial improvement with FlashBack, while the other tasks do not. Combining the limited improvements (sometime worse) performance on other tasks (appendix B1), there is no strong evidence on the effectiveness of FlashBack. Similar observations are made on StreamingBench. Besides, the authors seemed to selectively report tasks in the main text while keeping a subset in Appendix. While this can be justified by the relevance of the tasks to the problem context, it is necessary to apply and report rigorous selection criteria on task selection, which are currently not stated clearly and explicitly.
- For experimental evaluation, the ablation study creates useful insights into the effectiveness of different design elements, which is good. However, it lacks in-depth investigation of the role of history and whether the proposed techniques adequately utilized the history. To show how the memory bank is used and enhances the outcome, it is important to use tasks where there is unambiguous association (if not causal) between the quality/validity of an answer to the presence of history, i.e., evidence that show the model used the correct history evidence. Figure 3 provided some evidence but stopped at the aggregated/overall Native vs. recall. In addition, the visualization cases are too brief to show convincing evidence.
Questions for authors
- How are the categories of the context defined (Eq.5)? What is the basis of their mapping to Recall/Native/Refine?
- What criteria are applied to the selection of tasks on the two benchmarks?
- Some training-free baselines show superior performance compared to FlashBack? Is it possible to do benchmarking on these backbones?
- Are there intermedia evaluation and results on the efficacy of routing, i.e. performance related to correct identification of the history vs. native information?
- Code not available on anonymous github.
9 comments
A (W2 & Q2):
Thank you for raising this concern. Our evaluation is organized at two complementary levels. Tables 4 and 6 report aggregate results for all three task groups of each benchmark in the main text: Realtime, Backward, and Forward on OVO-Bench, and Real-Time Visual Understanding, Omni-Source Understanding, and Contextual Understanding on StreamingBench. Representative tasks are then selected to cover complementary temporal-evidence requirements and explain how memory access affects different forms of temporal reasoning.
The improvements extend beyond CRR. On Mage-VL, FlashBack improves REC from 19.48% to 29.80% and ASI from 58.78% to 64.86% on OVO-Bench. On StreamingBench, it improves CT from 40.43% to 46.28% and EU from 76.58% to 81.01%. These results demonstrate benefits across repeated-event tracking, action ordering, counting, and event understanding.
On OVO-Bench, Table 5 examines five complementary evidence requirements: recalling a previously observed event or state (EPM), preserving the order of completed actions (ASI), relating current events to earlier repetitions (REC), accumulating clues before answering (CRR), and rejecting questions unsupported by video evidence (HLD). Including HLD provides a contrast between recovering supporting historical evidence and rejecting an unsupported premise. These tasks illustrate both the benefits of historical access and the potential interference introduced by additional context.
On StreamingBench, Table 7 presents representative tasks involving temporal context, historical retention, and evidence accumulation. CR evaluates causal relations between events; CS and EU involve summarizing or understanding information across temporal segments; CT requires accumulating repeated observations; and ACU uses surrounding context to identify anomalous situations. These tasks are included to examine complementary forms of temporal integration. Their evidence requirements cut across the benchmark group labels: for example, CT belongs to Real-Time Visual Understanding but requires accumulating observations over time. Across the tasks in Table 7, performance remains stable or improves on both backbones.
Complete task-level results are provided in Appendix B: Table 9 for OVO-Bench and Tables 10–11 for StreamingBench. Performance decreases are also discussed in the main text, including HLD in Table 5 and Contextual Understanding on Mage-VL in Table 6. Together, the group-level comparisons establish the overall performance profile, while the task-level analyses characterize the benefits and trade-offs of memory access across different temporal-evidence requirements.
A (W3 & Q4):
Thank you for raising this important point. We analyze memory utilization through three complementary aspects: trajectory selection, historical reach, and the correspondence between retrieved visual evidence and answering outcomes. Since our evaluation uses answer-level annotations rather than gold page-level evidence annotations, we combine outcome-based evaluation of fixed Native/Recall trajectories with qualitative inspection of retrieved pages, questions, and answers. This also accommodates cases where supporting evidence is indirect or distributed across multiple observations.
First, Appendix A.5 quantitatively evaluates trajectory selection on StreamingVLM. On CRR, 57 queries have a correct Recall output and an incorrect Native output, while 11 show the reverse pattern, demonstrating the complementary capabilities of the two trajectories. On queries where exactly one trajectory is correct, Structured Routing achieves unique-correct selection rates of 65.91% on MA and 83.82% on CRR, compared with 45.45% and 16.18% for Direct Routing.
Second, Figure 3 quantifies the temporal reach of historical retrieval. On OVO-Bench, the farthest retrieved historical units originate 24 min 13 s before the query for Mage-VL and 21 min 39 s for StreamingVLM. On StreamingBench, the corresponding query-relative distances are 16 min 02 s and 12 min 54 s, respectively. These results show that Recall accesses history well beyond the bounded recent context.
Third, Figures 5 and 6 connect retrieved visual evidence to the resulting answers through successful examples. In Figure 5, for the query at 325 s, Mage-VL retrieves the 245.5–249.5 s page showing beer being poured into the flour mixture and stirred, supporting the correct answer to the action-sequence question. For the vehicle-color query at 66 s, StreamingVLM retrieves the 0–4 s page showing a blue car and answers Blue, whereas Native answers Black.
Figure 6 provides two additional action-sequence examples: Mage-VL retrieves the preparation action at 157.7–161.7 s preceding the division of dough into smaller balls, while StreamingVLM retrieves the 152–156 s segment showing the steak being placed on the grill before the subsequent flipping action. In these cases, the retrieved historical content visibly supports the correct answers.
Figure 7 presents a contrasting failure case. Native correctly identifies the vehicle as a Sedan, whereas Recall predicts SUV. The retrieved 4–8 s page contains the queried vehicle, but the Recall trajectory makes an incorrect fine-grained vehicle-type judgment. This illustrates the distinction between retrieving relevant historical content and correctly interpreting it during answering.
Together, these analyses connect when to recall, what historical evidence is accessed, and how that evidence is used for answering. FlashBack uses semantic routing to determine the evidence-access trajectory, query-dependent retrieval to select historical units from the causal prefix, and query-local Side-KV to make that evidence available to the frozen backbone. The successful and failure cases demonstrate that retrieving relevant historical evidence and correctly utilizing it during answering are distinct capabilities. We will further emphasize these connections between queries, retrieved pages, and resulting answers in the revised manuscript and appendix.
Thank you for your thorough review and constructive feedback. We have carefully considered your comments and provide our detailed responses below.
A (W1 & Q1):
Thank you for raising these important questions. The central insight of FlashBack is that the temporal-evidence understanding already present in a frozen VLM can be operationalized as an explicit memory-access policy. FlashBack uses the frozen VLM to infer what evidence a question requires and translates that understanding into a choice between continuing the Native streaming trajectory and accessing historical evidence through an isolated Recall trajectory. This connects existing semantic capability to memory control without additional training.
The five temporal-evidence states express the temporal reference and composition of the requested evidence: Current concerns a query-boundary state or activity; Historical concerns an explicitly earlier bounded event or state; Interaction concerns observed temporal relations within a bounded episode, including ordering, transitions, changes, interactions, and causal relations; Aggregation concerns statistics, coverage judgments, or cross-period comparisons over multiple moments in the observed video prefix; and Predictive concerns an unobserved future outcome. These are shared semantic evidence roles across backbones and benchmarks. Appendix A.4 provides their definitions and the precedence rules for assigning a single operational state.
A particularly important insight is that the time referred to by a question and the time span of evidence needed to answer it can differ. A question about the present may require recovering a state established earlier, rather than simply inspecting the current observation. This motivates the second-stage refinement of Current: Boundary and Ongoing map to Native, whereas Persistent and Outcome map to Recall to recover the evidence establishing a continuing state or realized result. Refine therefore resolves the evidence dependency before selecting one of the same two answering trajectories. Historical, Interaction, and Aggregation invoke Recall to reconstruct historical or cross-time evidence, while Predictive continues Native inference from the available causal streaming state.
The many-to-one semantic-to-action mapping also limits error propagation: misclassification among semantic states mapped to the same final action does not change the selected trajectory. Only errors that cross the Native/Recall decision boundary can alter trajectory selection.
Representation-balanced inference serves as an auxiliary mechanism for stabilizing these semantic decisions; its details are provided in Appendix A.4.
Appendix A.5 provides empirical support for this semantic-to-action interface on StreamingVLM. Using identical fixed Native and Recall outputs, Structured Routing is compared with Direct Routing, which predicts Native or Recall without the intermediate temporal-evidence decomposition. On queries where exactly one trajectory is correct, Structured Routing achieves unique-correct selection rates of 65.91% on MA and 83.82% on CRR, compared with 45.45% and 16.18% for Direct Routing. These results support the effectiveness of explicitly structuring the temporal-evidence understanding of the frozen VLM to enable selective memory access.