acceptodds
Under review as a conference paper at ICLR 2027

FlashBack: Knowing When to Remember in Streaming Vision-Language Models

Abstract

Streaming vision-language models must process continuously growing video streams under a bounded compute budget, creating a persistent tension between real-time perception and long-term memory. Retrieving historical information provides a natural remedy, yet historical recall is not uniformly beneficial: unnecessary history may introduce irrelevant context into current reasoning and interfere with native real-time perception. Effective streaming memory should therefore address not only what to remember, but also when and how to access it. To this end, we introduce FlashBack, a training-free framework for selective, multi-level memory in streaming vision-language models. Before retrieving history, FlashBack draws on the semantic understanding of the frozen streaming VLM to infer whether a query calls for historical evidence. This assessment determines whether inference remains on the Native trajectory or invokes an isolated Recall trajectory. The Recall trajectory combines recent context with retrieved long-term memory through a query-local Side-KV pathway, preserving local temporal continuity without modifying the persistent Native state. We instantiate FlashBack on StreamingVLM and Mage-VL-4B and evaluate it on OVO-Bench and StreamingBench. The results show improvements on several long-horizon and memory-dependent tasks while largely preserving real-time perception, with performance competitive with strong training-based streaming methods despite requiring no additional training. Our code is released at https://anonymous.4open.science/status/Flashback-8D52 .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Vision-Language Models, FlashBack, Selective Multi-level Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XttEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a method, FlashBack, to deal with the utilization of historical information in streaming video QA. It tries to enhance the ability of VLMs to make use of such history via a deliberately designed mechanism - to activate the route of history recall vs. current state. The authors reported positive outcomes of FlashBack using two VLM backbones (StreamingVLM, Mega-VL) on selected tasks of two benchmarks (SteamingBench and OvoBench).

Critical strengths

The paper is well-motivated on using history to answer visual questions in long videos. A training-free mechanism with frozen-VLM is notable and valid. The overall process is clearly explained and plausible. The design of experiment is in general valid by including multiple (2) VLMs and on benchmarks (2). The ablation study is also relevant to attribute the effect to the components, so as to explain the underlying mechanism.

Critical weaknesses
  1. The methodology – FlashBack- while well-motivated and clearly presented, could be brittle by adopting a layered/sequential process that separates the inferencing into temporal evidence routing and recall (answer generation). Specifically, for temporal evidence routing, Eq. 5 proposes a few categories of context space and their dispatching to different semantic-action mapping. First, there is no justification of the basis of these context categories and whether they are mutually exclusive and self-containing. There is neither reported nor empirical evidence on the design of these categories. Second, during inference stage, if the classification is wrong, the error will propagate thus voiding subsequent recall. This can be partially shown by the small margin (sometime mixed results) of the proposed approach overall. Last, the mechanism of “know when to remember”, which is a central component of the methodology, shows limited technical merit by using a naïve process of prompting and aggregating the response. Therefore, the novelty and significance of the work is limited.
  2. The main results show limited advantage of the proposed approach. While I appreciated the author’s honesty in reporting the results, the margin of FlashBack is very small and sometimes inferior to the benchmarks. On OVO-Bench, compared with the non-FlashBack baselines, only CRR exhibited substantial improvement with FlashBack, while the other tasks do not. Combining the limited improvements (sometime worse) performance on other tasks (appendix B1), there is no strong evidence on the effectiveness of FlashBack. Similar observations are made on StreamingBench. Besides, the authors seemed to selectively report tasks in the main text while keeping a subset in Appendix. While this can be justified by the relevance of the tasks to the problem context, it is necessary to apply and report rigorous selection criteria on task selection, which are currently not stated clearly and explicitly.
  3. For experimental evaluation, the ablation study creates useful insights into the effectiveness of different design elements, which is good. However, it lacks in-depth investigation of the role of history and whether the proposed techniques adequately utilized the history. To show how the memory bank is used and enhances the outcome, it is important to use tasks where there is unambiguous association (if not causal) between the quality/validity of an answer to the presence of history, i.e., evidence that show the model used the correct history evidence. Figure 3 provided some evidence but stopped at the aggregated/overall Native vs. recall. In addition, the visualization cases are too brief to show convincing evidence.
Questions for authors
  1. How are the categories of the context defined (Eq.5)? What is the basis of their mapping to Recall/Native/Refine?
  2. What criteria are applied to the selection of tasks on the two benchmarks?
  3. Some training-free baselines show superior performance compared to FlashBack? Is it possible to do benchmarking on these backbones?
  4. Are there intermedia evaluation and results on the efficacy of routing, i.e. performance related to correct identification of the history vs. native information?
  5. Code not available on anonymous github.
9 comments

Thank you for your thorough review and constructive feedback. We have carefully considered your comments and provide our detailed responses below.

A (W1 & Q1):

Thank you for raising these important questions. The central insight of FlashBack is that the temporal-evidence understanding already present in a frozen VLM can be operationalized as an explicit memory-access policy. FlashBack uses the frozen VLM to infer what evidence a question requires and translates that understanding into a choice between continuing the Native streaming trajectory and accessing historical evidence through an isolated Recall trajectory. This connects existing semantic capability to memory control without additional training.

The five temporal-evidence states express the temporal reference and composition of the requested evidence: Current concerns a query-boundary state or activity; Historical concerns an explicitly earlier bounded event or state; Interaction concerns observed temporal relations within a bounded episode, including ordering, transitions, changes, interactions, and causal relations; Aggregation concerns statistics, coverage judgments, or cross-period comparisons over multiple moments in the observed video prefix; and Predictive concerns an unobserved future outcome. These are shared semantic evidence roles across backbones and benchmarks. Appendix A.4 provides their definitions and the precedence rules for assigning a single operational state.

A particularly important insight is that the time referred to by a question and the time span of evidence needed to answer it can differ. A question about the present may require recovering a state established earlier, rather than simply inspecting the current observation. This motivates the second-stage refinement of Current: Boundary and Ongoing map to Native, whereas Persistent and Outcome map to Recall to recover the evidence establishing a continuing state or realized result. Refine therefore resolves the evidence dependency before selecting one of the same two answering trajectories. Historical, Interaction, and Aggregation invoke Recall to reconstruct historical or cross-time evidence, while Predictive continues Native inference from the available causal streaming state.

The many-to-one semantic-to-action mapping also limits error propagation: misclassification among semantic states mapped to the same final action does not change the selected trajectory. Only errors that cross the Native/Recall decision boundary can alter trajectory selection.

Representation-balanced inference serves as an auxiliary mechanism for stabilizing these semantic decisions; its details are provided in Appendix A.4.

Appendix A.5 provides empirical support for this semantic-to-action interface on StreamingVLM. Using identical fixed Native and Recall outputs, Structured Routing is compared with Direct Routing, which predicts Native or Recall without the intermediate temporal-evidence decomposition. On queries where exactly one trajectory is correct, Structured Routing achieves unique-correct selection rates of 65.91% on MA and 83.82% on CRR, compared with 45.45% and 16.18% for Direct Routing. These results support the effectiveness of explicitly structuring the temporal-evidence understanding of the frozen VLM to enable selective memory access.

A (W2 & Q2):

Thank you for raising this concern. Our evaluation is organized at two complementary levels. Tables 4 and 6 report aggregate results for all three task groups of each benchmark in the main text: Realtime, Backward, and Forward on OVO-Bench, and Real-Time Visual Understanding, Omni-Source Understanding, and Contextual Understanding on StreamingBench. Representative tasks are then selected to cover complementary temporal-evidence requirements and explain how memory access affects different forms of temporal reasoning.

The improvements extend beyond CRR. On Mage-VL, FlashBack improves REC from 19.48% to 29.80% and ASI from 58.78% to 64.86% on OVO-Bench. On StreamingBench, it improves CT from 40.43% to 46.28% and EU from 76.58% to 81.01%. These results demonstrate benefits across repeated-event tracking, action ordering, counting, and event understanding.

On OVO-Bench, Table 5 examines five complementary evidence requirements: recalling a previously observed event or state (EPM), preserving the order of completed actions (ASI), relating current events to earlier repetitions (REC), accumulating clues before answering (CRR), and rejecting questions unsupported by video evidence (HLD). Including HLD provides a contrast between recovering supporting historical evidence and rejecting an unsupported premise. These tasks illustrate both the benefits of historical access and the potential interference introduced by additional context.

On StreamingBench, Table 7 presents representative tasks involving temporal context, historical retention, and evidence accumulation. CR evaluates causal relations between events; CS and EU involve summarizing or understanding information across temporal segments; CT requires accumulating repeated observations; and ACU uses surrounding context to identify anomalous situations. These tasks are included to examine complementary forms of temporal integration. Their evidence requirements cut across the benchmark group labels: for example, CT belongs to Real-Time Visual Understanding but requires accumulating observations over time. Across the tasks in Table 7, performance remains stable or improves on both backbones.

Complete task-level results are provided in Appendix B: Table 9 for OVO-Bench and Tables 10–11 for StreamingBench. Performance decreases are also discussed in the main text, including HLD in Table 5 and Contextual Understanding on Mage-VL in Table 6. Together, the group-level comparisons establish the overall performance profile, while the task-level analyses characterize the benefits and trade-offs of memory access across different temporal-evidence requirements.

A (W3 & Q4):

Thank you for raising this important point. We analyze memory utilization through three complementary aspects: trajectory selection, historical reach, and the correspondence between retrieved visual evidence and answering outcomes. Since our evaluation uses answer-level annotations rather than gold page-level evidence annotations, we combine outcome-based evaluation of fixed Native/Recall trajectories with qualitative inspection of retrieved pages, questions, and answers. This also accommodates cases where supporting evidence is indirect or distributed across multiple observations.

First, Appendix A.5 quantitatively evaluates trajectory selection on StreamingVLM. On CRR, 57 queries have a correct Recall output and an incorrect Native output, while 11 show the reverse pattern, demonstrating the complementary capabilities of the two trajectories. On queries where exactly one trajectory is correct, Structured Routing achieves unique-correct selection rates of 65.91% on MA and 83.82% on CRR, compared with 45.45% and 16.18% for Direct Routing.

Second, Figure 3 quantifies the temporal reach of historical retrieval. On OVO-Bench, the farthest retrieved historical units originate 24 min 13 s before the query for Mage-VL and 21 min 39 s for StreamingVLM. On StreamingBench, the corresponding query-relative distances are 16 min 02 s and 12 min 54 s, respectively. These results show that Recall accesses history well beyond the bounded recent context.

Third, Figures 5 and 6 connect retrieved visual evidence to the resulting answers through successful examples. In Figure 5, for the query at 325 s, Mage-VL retrieves the 245.5–249.5 s page showing beer being poured into the flour mixture and stirred, supporting the correct answer to the action-sequence question. For the vehicle-color query at 66 s, StreamingVLM retrieves the 0–4 s page showing a blue car and answers Blue, whereas Native answers Black.

Figure 6 provides two additional action-sequence examples: Mage-VL retrieves the preparation action at 157.7–161.7 s preceding the division of dough into smaller balls, while StreamingVLM retrieves the 152–156 s segment showing the steak being placed on the grill before the subsequent flipping action. In these cases, the retrieved historical content visibly supports the correct answers.

Figure 7 presents a contrasting failure case. Native correctly identifies the vehicle as a Sedan, whereas Recall predicts SUV. The retrieved 4–8 s page contains the queried vehicle, but the Recall trajectory makes an incorrect fine-grained vehicle-type judgment. This illustrates the distinction between retrieving relevant historical content and correctly interpreting it during answering.

Together, these analyses connect when to recall, what historical evidence is accessed, and how that evidence is used for answering. FlashBack uses semantic routing to determine the evidence-access trajectory, query-dependent retrieval to select historical units from the causal prefix, and query-local Side-KV to make that evidence available to the frozen backbone. The successful and failure cases demonstrate that retrieving relevant historical evidence and correctly utilizing it during answering are distinct capabilities. We will further emphasize these connections between queries, retrieved pages, and resulting answers in the revised manuscript and appendix.

Reviewer_BmcJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlashBack, a training-free method that builds a two-level memory for frozen streaming VLMs. A two-stage router only uses the question text to choose the Native or Recall trajectory. Recall answers with one retrieved historical page and the recent context in a query-local Side-KV, which is not written back. On OVO-Bench, CRR and REC improve on both StreamingVLM-7B and Mage-VL-4B, but HLD decreases. The authors claim that FlashBack largely preserves real-time perception and achieves competitive performance with training-based methods.

Critical strengths
  1. The idea of separating whether to recall from what to recall is clear, and Side-KV keeps the Native state unchanged. This is the most valuable part of the paper.
  2. FlashBack is training-free and can be applied to two different streaming backbones. Its gains also appear outside the ablation subsets, which makes the results more convincing.
Critical weaknesses
  1. In Tables 1 and 2, K=0 uses no historical page but already improves CRR over Native from 41.67 to 55.42. Therefore, most of the CRR gain comes from the Side-KV with only recent context, and the historical page only adds 5.41 points. This weakens the main claim about long-term memory.
  2. Compared with Always Recall in Table 8, Structured Routing has the same CRR and is only 6 questions better on MA. Since Always Recall is not reported on the full benchmarks, the router's contribution cannot be verified. As the router is the key idea of the title, this also lowers my rating.
Questions for authors
  1. Why does K=0 already outperform Native? Does the difference come from the attention sinks, the accumulated text context, or the visual window in the Native state?
  2. It would be better to report Always Recall and K=0 on the full OVO-Bench and StreamingBench.
  3. The three variants in Table 3 also do not write back, so a write-back variant should be added to test the No Write-back design in Section 3.4.
  4. What is the per-query latency of Recall, and how large is the memory bank on long videos?
9 comments

Thank you for your careful evaluation and helpful suggestions. We have addressed your questions through additional experiments and analyses. Our detailed responses are provided below.

A (W1 & Q1):

Thank you for raising this important question. To investigate the gain at K=0, we conducted additional controlled experiments on StreamingVLM-7B. Both FlashBack configurations below use the same Structured Routing policy. For queries routed to Recall, K=0 uses only the recent context, and K=1 additionally retrieves one eligible historical page when available.

ConfigurationCRR accuracy (%)
Native41.67
FlashBack (K=0)55.42
FlashBack (K=1)60.83

Native and K=0 use the same recent visual observations, sampled at 2 FPS within a window of up to 16 seconds. Their difference lies in how these observations are processed by the language decoder.

With the Native visual settings and answer scoring fixed, we tested three changes separately: shortening the text history, removing the attention sinks associated with preceding text, and removing all preceding text content. CRR accuracy remained between 41.25% and 41.67% across these controls.

The K=0 result highlights a broader insight: memory effectiveness depends not only on what information is retained, but also on how that information is made available for reasoning. FlashBack translates this insight into a separation between persistent streaming-state maintenance and query-local evidence readout. Native answers using recent visual KV states computed incrementally along the streaming trajectory. The text-history controls modify the retained text context while preserving these already-computed visual states. In contrast, Recall at K=0 reconstructs an isolated decoder context from saved visual embeddings, allowing the same recent observations to participate in answering through freshly constructed decoder states. FlashBack therefore provides more than access to additional history: it also provides an alternative reasoning context for evidence already retained by the streaming system. With the same recent visual input and frozen model weights, this query-local reconstruction improves CRR from 41.67% to 55.42%, a gain of 13.75 percentage points.

Long-term retrieval provides a complementary contribution. Under the same routing policy, adding one historical page further raises accuracy from 55.42% to 60.83%, an additional gain of 5.41 percentage points. Together, these results distinguish two complementary contributions of FlashBack: query-local reconstruction of retained recent evidence and retrieval of additional long-term historical evidence.

A (W2 & Q2):

Thank you for raising this concern. We added full-benchmark comparisons on OVO-Bench and StreamingBench for both backbones using fixed branch outputs from our causal streaming evaluations. Always Recall uses the fixed K=1 Recall outputs, which Structured Routing at K=1 uses whenever it selects Recall. The two Structured settings share identical routing decisions and Native outputs. At K=0, the Recall branch uses only recent context; K=1 additionally retrieves one eligible historical page when available. All comparisons use the same answer-scoring protocol.

BackboneStrategyRealtimeBackwardForwardOVO Avg.StreamingBench Overall
Mage-VLAlways Recall (K=1)76.4649.6252.1559.4171.12
Mage-VLStructured (K=0)81.4748.9150.6660.3574.60
Mage-VLStructured (K=1)81.2049.9351.9461.0274.67
StreamingVLMAlways Recall (K=1)69.4246.4745.8353.9167.26
StreamingVLMStructured (K=0)70.9143.7339.4651.3668.13
StreamingVLMStructured (K=1)70.8544.9241.2252.3367.66

On Mage-VL, Structured Routing at K=1 improves OVO Avg. from 59.41 to 61.02 and StreamingBench Overall from 71.12 to 74.67, gains of 1.61 and 3.55 percentage points, respectively. Realtime accuracy improves by 4.74 points, while Backward and Forward performance remains close to Always Recall. These results demonstrate that selective memory access improves overall accuracy and real-time performance while retaining the benefits of Recall for history-dependent reasoning. Across both backbones, adding a historical page also improves Backward and Forward accuracy over the corresponding K=0 setting.

On StreamingVLM, Structured Routing at K=1 improves Realtime accuracy by 1.43 points and StreamingBench Overall by 0.40 points over Always Recall. Its OVO Avg. is 1.58 points lower, with SSR accounting for approximately 1.54 points under the official group macro-average. SSR asks whether a specified procedural step is currently being performed, and Structured Routing selects Native for these queries. StreamingVLM achieves 42.77% with Native and 56.60% with Always Recall.

The cross-backbone contrast highlights the importance of the underlying Native trajectory. StreamingVLM is adapted for continuous narration through overlapping-chunk supervision and persistent KV reuse, whereas Mage-VL already processes recent visual context at query time. On SSR, the same temporal-evidence policy selects Native on both backbones, yet the Native–Always Recall difference is only 0.64 points on Mage-VL, compared with 13.83 points on StreamingVLM. Moreover, an additional diagnostic that forces recent-only Recall on all SSR queries reaches 58.98% with the same frozen StreamingVLM backbone. This shows that recent-context reconstruction alone can account for the SSR improvement, without requiring longer-term historical retrieval. The gap exposes a task-specific distinction between retaining recent visual information and making that information effective for query-time reasoning within the Native StreamingVLM state. These results highlight that realizing the benefits of semantic memory control also depends on how effectively the underlying Native trajectory contextualizes retained evidence for answering.

Appendix A.5 further provides a direct evaluation of trajectory selection on StreamingVLM using fixed Native and Recall outputs. Among queries where exactly one trajectory is correct, Structured Routing achieves unique-correct selection rates of 65.91% on MA and 83.82% on CRR, compared with 45.45% and 16.18% for Direct Routing. Together, the full-benchmark comparisons and fixed-output analysis demonstrate that structured temporal-evidence reasoning improves trajectory selection over direct Native/Recall prediction and enables effective selective memory access without additional training.

A (Q3):

Thank you for this helpful suggestion. To directly evaluate the No Write-back design, we conducted additional multi-query experiments on both backbones. Queries were processed in chronological order within each video stream. We compared Write-back with No Write-back while keeping the Native state management, routing, retrieval, and answer-scoring protocol unchanged. The only intervention is whether the actual KV states produced by a Recall forward are retained for subsequent Native inference.

For StreamingVLM, Write-back retains the Recall KV states as additional attention context for subsequent Native processing, while preserving the original Native cache and its maintenance mechanism. Retained visual states are evicted when their age reaches four video-observation blocks. Retained text and other states follow a sixteen-round Native lifetime, after which the entire retained segment is discarded. Both age counters start when the Recall KV states are committed for cross-query retention. For Mage-VL, the most recent Recall KV states are retained as additional attention context for subsequent Native queries, replaced by the next Recall, and cleared at the end of the stream. Under No Write-back, temporary Recall KV states are discarded after the current query, while archived historical evidence remains available for future retrieval.

BackboneStrategyMACRRMCUCS
StreamingVLMWrite-back70.0059.5842.8088.33
StreamingVLMNo Write-back70.4059.5842.8088.33
Mage-VLWrite-back87.2059.1756.4089.27
Mage-VLNo Write-back90.4059.1763.2090.54

Based on the accuracies shown in the table, No Write-back maintains comparable performance on StreamingVLM and improves MA, MCU, and CS on Mage-VL by 3.20, 6.80, and 1.27 percentage points, respectively. These results support keeping Recall states query-local: historical evidence remains available for future retrieval, while the temporary decoder states assembled for one Recall query do not become additional persistent conditioning for subsequent Native inference.

Reviewer_YAJbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FlashBack, a training-free module that can be directly applied to Streaming VLMs without additional training. It balances short-term continuity and long-term memory, and supports on-demand access instead of retrieving historical information every time. It achieves promising performance across multiple benchmarks and is compatible with several mainstream Streaming VLMs.

Critical strengths
  1. This paper proposes a training-free method, which is validated on two different streaming backbones. It features controllable cost, strong applicability, and brings certain performance gains for long-range temporal tasks.
  2. Side-KV provides a clean solution to state contamination. It retrieves historical information when necessary and avoids introducing historical information as interference when it is not needed. As a lightweight module, it eliminates interference and brings a certain improvement in efficiency.
Critical weaknesses
  1. The method proposed in this paper yields concerning experimental results. On the OVO-Bench benchmark, it reduces the HLD accuracy of StreamingVLM, and in StreamingBench, the context understanding score of Mage-VL also decreases. The authors need to provide valid explanations for these aspects.
  2. This paper proposes a training-free method. It needs to be validated on multiple different mainstream streaming VLMs and various benchmarks to verify its generalization ability and demonstrate the advantages of the proposed method. However, shortcomings still exist at present.
Questions for authors

See Weaknesses

6 comments

Thank you for your thoughtful review and constructive comments. Our detailed responses to your concerns are provided below.

A (W1):

Thank you for highlighting these performance decreases. We view them as informative observations for streaming and long-context video understanding. They suggest a shared challenge: semantically relevant history is not necessarily valid evidence for the question being answered. A router may identify the need for historical information, while the retrieved content still requires verification against the specific event and time referred to by the question. The distinction is therefore between accessing potentially relevant history and determining whether that history warrants the answer.

Standard historical questions and HLD questions involving events that did not occur can have similar surface forms. For example, “Where did he place the cup earlier?” may refer to an actual event or presuppose an event that never occurred. From the question alone, the router can identify the need for historical evidence, but this does not establish whether the event actually occurred. As discussed in §4.4, retrieved history can be temporally or semantically related to a query without constituting valid evidence for its premise. The model may therefore mistake related historical content for support of an unsupported premise, making such questions harder to reject. This is consistent with the HLD decreases in Table 5: from 27.96 to 23.12 for StreamingVLM and from 32.26 to 31.72 for Mage-VL. This suggests that historical retrieval must be accompanied by event-specific premise verification: content resembling the queried event should not, by itself, be treated as evidence that the event occurred.

Table 11 shows that the group average decreases from 62.13 to 61.07, entirely due to MCU (64.80→61.60), while ACU and SQA remain unchanged. MCU requires distinguishing the current state from similar cues that appeared earlier but no longer reflect the current situation. Historical information can therefore be factually correct yet introduce outdated context that competes with evidence about the current scene. The decrease is consistent with the challenge of suppressing obsolete historical cues when answering current-state questions. This points to a complementary requirement of temporal validity: even factually correct historical information may need to be suppressed when it no longer describes the state being queried.

Taken together, these results highlight a broader requirement for reliable streaming and long-context reasoning: models must determine not only which historical information to retrieve, but also which retrieved information should influence the answer. In HLD, this requires verifying whether retrieved content supports the specific event premise; in MCU, it requires assessing whether historical information remains valid for the current state. FlashBack addresses when and how historical evidence is accessed through query-dependent routing and Side-KV, while these observations point to evidence verification and temporal suppression as complementary capabilities for reliable memory-augmented reasoning.

A (W2):

Thank you for raising the concern about generalization. FlashBack provides a shared memory-access mechanism across different streaming architectures. We evaluate this transfer through two architecturally distinct native streaming VLMs and two complementary benchmark suites, and further supplement these experiments with a matched-backbone comparison on a standard frozen Qwen checkpoint.

StreamingVLM-7B and Mage-VL-4B organize their Native states differently. StreamingVLM maintains a bounded persistent decoder KV state, whereas Mage-VL maintains a rolling visual context and performs query-time decoding over a local visual window within its dual-system architecture. FlashBack preserves the original Native state organization and maintenance rules in both cases, while applying the same temporal-evidence definitions, Native/Recall decision rules, and query-local memory-isolation principle. This demonstrates that the same memory-control mechanism can operate across distinct Native state organizations, without additional training or backbone-specific parameter updates.

OVO-Bench and StreamingBench provide complementary evaluation coverage. OVO-Bench evaluates Realtime, Backward, and Forward reasoning, while StreamingBench covers Real-Time Visual Understanding, Omni-Source Understanding, and Contextual Understanding. On OVO-Bench, overall accuracy improves from 50.93 to 52.33 for StreamingVLM and from 58.27 to 61.02 for Mage-VL. On StreamingBench, Real-Time Visual Understanding improves from 78.82 to 79.22 and from 81.63 to 82.59, respectively.

We additionally compare FlashBack with HERMES using the same frozen Qwen2.5-VL-7B-Instruct checkpoint. Both methods use 0.5 FPS input, 16-frame processing chunks, a 6,000-token online visual KV budget, and the same candidate-logit answer-scoring protocol. Following the evaluation scope released with HERMES, FlashBack achieves 59.03 versus 56.80 on the OVO-Bench Realtime/Backward average and 77.07 versus 76.83 on StreamingBench Realtime. This adds a matched-checkpoint evaluation beyond the two native-streaming instantiations.

Together, these experiments support the portability of FlashBack across different Native state organizations and backbone settings. They show that selective memory access can improve temporal reasoning while largely preserving or improving real-time accuracy without additional training.

We will incorporate the corresponding revisions, additional experimental results, and analyses into the main text and appendices of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.