- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper addresses online memory construction for continuous video streams, aiming to make historical observations searchable while keeping pace with incoming video. It proposes LoreMem, a training-free method that maps visual observations and dialogue to a fixed lexicon and stores sparse concept identities and ranks. At query time, these ranks are used to retrieve original video segments for answering. Experiments evaluate retrieval and QA quality on four long-form video benchmarks, alongside construction workload using measured mobile-device processing times.
Critical strengths
S1: The proposed method effectively addresses the need to construct useful, searchable memories as video continues to arrive.
Its lightweight indexing supports retrieval over extended video histories, improving QA accuracy across four long-form video benchmarks. Mobile-device measurements further support its ability to keep pace with incoming segments without accumulating construction backlog.
Critical weaknesses
W1: The comparisons lack representative streaming models and memory mechanisms relevant to the paper’s motivation.
Although the evaluated benchmarks assess long-term video memory, the baselines mainly cover direct QA, caption memory, and RAG systems. Comparisons omit methods such as Flash-VStream, Dispider, Memento, TimeChat-Online, StreamForest, VST, and JoyAI-VL-Interaction, which offer alternative ways to maintain and use historical information as video arrives. Comparing a representative subset under fair conditions is necessary to clarify LoreMem’s advantages over existing online memory approaches.
W2: This method is limited in its ability to model temporal and contextual relationships.
Each segment is indexed using only its own content, without explicitly linking preceding events. In my opinoin, expressions such as “there are still eight left” and “the three-eyed guy” in Figure 10 highly depend on context, once losing their referents, it causes retrieval failure. Even Figure 9 may become challenging if asked “What did I eat for takeout last night?”: similar meal segments may require surrounding nighttime/daylight cues to distinguish them, but those contextual segments may not be retrieved. Missing context at retrieval cannot be recovered during answering.
References:
[R1] Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
[R3] Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video
[R4] TimeChat-Online: 80% Visual Tokens Are Naturally Redundant in Streaming Videos
[R5] StreamForest: Efficient Online Video Understanding with Persistent Event Memory
[R6] Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
[R7] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Questions for authors
Q1: Does the proposed method consistently select relevant historical evidence?
Figures 7–10 show only the highest-ranked concepts in ground-truth evidence segments. Could the authors visualize the complete retrieval process, including the concepts selected by the query, their ranks in candidate segments, the final segment scores, and the actual retrieved segments and frames?
Q2: Do high-frequency, generic concepts dominate retrieval scores, making it difficult to distinguish similar segments from different events or causing certain segments to be retrieved frequently?
Could the authors list the most frequent concepts in the Top-M indexes and their frequencies, and report how often each segment is retrieved? Examples comparing concept rankings across repeated activities would also help assess whether the representation is sufficiently discriminative.
Q3: Section 5.1 restricts No-Memory baselines to the latest 300 frames, while retrieval methods select 300 frames from the eligible history. Does this strategy restrict the original models’ capabilities?
Could models supporting full-video input or historical compression access the entire history through their native mechanisms, with accuracy, runtime, and memory costs reported together?
Q4: The proposed method relies heavily on the chosen lexicon. How were the 65,383 WordNet entries selected?
Beyond varying the number of retained concepts, have the authors evaluated whether the underlying lexicon sufficiently covers their video content?
15 comments
Author's Response to Reviewer 9d8G (2/6)
W2: This method is limited in its ability to model temporal and contextual relationships.
Each segment is indexed using only its own content, without explicitly linking preceding events. In my opinoin, expressions such as “there are still eight left” and “the three-eyed guy” in Figure 10 highly depend on context, once losing their referents, it causes retrieval failure. Even Figure 9 may become challenging if asked “What did I eat for takeout last night?”: similar meal segments may require surrounding nighttime/daylight cues to distinguish them, but those contextual segments may not be retrieved. Missing context at retrieval cannot be recovered during answering.
Thank you for highlighting these context-dependent cases. LoreMem retains temporal metadata and pointers to the original video and dialogue alongside each segment's semantic index. The stored concept ranks provide retrieval cues, while the original observations remain available for contextual inspection. The context preserved in memory and the context selected by retrieval are therefore distinct.
For time-qualified questions such as “last night,” segment timestamps and the question time provide explicit temporal information. The EgoLifeQA/Ego-R1 answer prompt includes both fields (Appendix F.1, Table 18). A retrieval policy could use this information to constrain the relevant time window, without depending exclusively on surrounding nighttime/daylight frames. Retaining this information alone does not ensure that the required segment is selected.
Alongside expressions such as “there are still eight left” and “the three-eyed guy,” a segment's visual and dialogue memories provide additional semantic cues. LoreMem aggregates the stored ranks of query-selected concepts across both modalities, allowing other cues to support retrieval when a particular expression has an unclear referent. Concepts with identical lookup ranks across candidates contribute equally to their costs, while differences in other concept ranks provide discrimination. Figure 5 shows that increasing query support from to improves nDCG@20 on all four benchmarks, supporting the use of multiple query concepts. Explicit cross-segment reference resolution remains a limitation of the current single-pass protocol. Figure 10 illustrates a mismatch between the return-related query concepts and the annotated claw-machine event; it does not isolate the effect of cross-event relationships.
An agentic retrieval policy could inspect neighboring segments or issue follow-up searches over the preserved observations before answering. Such policies could retain LoreMem's segment-local construction procedure while adding query-time computation. This separation is consistent with our focus on building reusable indexes before future questions are known and keeping construction workload bounded under sustained arrivals. Appendix E of the revised manuscript now states the absence of explicit cross-segment relationships and discusses these complementary query-time mechanisms as possible extensions.
Author's Response to Reviewer 9d8G (3/6)
Q1: Does the proposed method consistently select relevant historical evidence?
Figures 7–10 show only the highest-ranked concepts in ground-truth evidence segments. Could the authors visualize the complete retrieval process, including the concepts selected by the query, their ranks in candidate segments, the final segment scores, and the actual retrieved segments and frames?
Thank you for requesting a trace from the query to the retrieved evidence. We have expanded Appendix D.7 and Figures 7–10 to show query-concept selection, candidate memory ranks and costs, final retrieval positions, and corresponding original observations. Panel (a) displays six representative concepts from the query's top- entries, with their query ranks and cosine similarities. Panel (b) reports candidate timestamps, visual and dialogue lookup ranks, costs, and retrieval positions; panel (c) provides frames and transcript excerpts. Figures 7–8 compare the top three candidates, while Figures 9–10 compare the top two candidates with the ground-truth segment, including its position when it falls outside the retrieved top 20.
The figures follow Section 4.4: concepts outside the retained top- memory receive the lookup rank , and retrieval selects the segments with the lowest fused cost, . The Cost (6) columns and their fused values are subtotals over the six illustrated concepts; the Cost (100) columns and final retrieval positions use all 100 query concepts. Appendix D.7 makes this distinction explicit, since the six-concept subtotals need not reproduce the full retrieval order.
The successful examples connect the retrieved segments to specific factual evidence. In Figure 7, the takeout segment is retrieved first: takeout ranks third in visual memory and second in dialogue memory, while the original dialogue identifies KFC roast duck and KFC pizza. Food-related alternatives are shown alongside it, allowing their concept support to be compared. In Figure 8, the coin-price segment is also retrieved first. Its dialogue memory ranks coin first and ego third, whereas the other displayed monetary discussions do not retain ego. The retrieved transcript supplies the precise answer, “each coin is now 3 RMB.”
Figures 9–10 expose different retrieval outcomes. In Figure 9, the annotated arcade segment ranks 9 and remains within the top-20 evidence set. Its visual memory ranks arcade, playing, and game highly, while its dialogue memory matches few of the displayed query concepts. In Figure 10, return-related query concepts match unrelated returning scenes; the annotated claw-machine segment ranks 876 and is excluded. This failure trace makes the mismatch between query-selected concepts and relevant evidence visible, alongside the successful cases.
These examples connect concept-level indexing to actual evidence inclusion or exclusion. Across benchmarks, Table 6 provides aggregate retrieval evidence: LoreMem achieves average temporal-evidence nDCG@20 of 18.8, compared with 7.2 for the strongest visual-embedding baseline and 9.7 for local caption memory. The expanded traces explain concrete retrieval outcomes and complement this measured retrieval utility.
Author's Response to Reviewer 9d8G (1/6)
We sincerely thank the reviewer 9d8G for their constructive feedback and recognition of the effectiveness of our proposed method in constructing searchable memories from continuously arriving video streams, its improved QA performance across four long-form video benchmarks, and its practical feasibility demonstrated by lightweight indexing and backlog-free operation on a mobile device. We address the reviewer's concerns in detail below.
Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.
Thank you for suggesting comparisons with streaming video-memory methods. We have evaluated all seven methods listed in the review: Flash-VStream, StreamForest, Dispider, VST, JoyAI-VL-Interaction (JoyAI), TimeChat-Online, and Memento. Their results are included in Appendix D.1 (Table 8), and their memory mechanisms are discussed in Section 2 of the revised manuscript.
The evaluation uses the 500-question EgoLifeQA subset reported in Appendix A.1 (Table 5). All methods are evaluated under the same input conditions, receiving video frames and aligned dialogue transcripts together with the same textual questions. Each system uses its released memory mechanism and corresponding answer model, with the Memento answerer replacement marked by † in the table.
LoreMem achieves 49.4% QA accuracy, exceeding all seven streaming baselines by 5.2–13.4 percentage points. Dispider obtains the strongest baseline result at 44.2%, followed by TimeChat-Online at 43.4% and Memento at 43.0%. LoreMem's gains over these methods are 5.2, 6.0, and 6.4 percentage points, respectively.
† marks the Memento answerer replacement with Qwen3-VL-8B, as specified in the caption of Table 8.
This comparison evaluates complete memory systems under shared input conditions, using their respective answer models. It complements the fixed-answerer history-selection control in Appendix D.1 (Table 7), where LoreMem and uniform sampling both use Qwen3-VL-8B and a 300-frame answer budget. LoreMem achieves 52.0% average QA accuracy compared with 41.3% for uniform sampling, a gain of 10.7 percentage points. Together, these comparisons support the utility of LoreMem's historical evidence selection under the evaluated settings.