acceptodds
Under review as a conference paper at ICLR 2027

Keeping Pace with Video: Online Multimodal Memory Construction

14 views

Abstract

Long-form video assistants must make incoming observations searchable during ongoing use, before future questions are known. This requires online memory construction that keeps pace with the video stream under a fixed compute budget. When construction falls behind, unfinished work accumulates and access to recent evidence is delayed. We formulate online memory construction as a joint quality-stability problem: supporting useful retrieval while keeping construction workload bounded under sustained arrivals. Building on the insight that linguistic indexing does not require generating descriptions, we introduce **LoreMem** (**L**exicon-**O**rdinal **Re**trieval **Mem**ory), a caption-free multimodal memory. Using frozen encoders,**LoreMem** selects concepts from a predefined lexicon to index visual observations and dialogue, storing sparse concept identities and ranks. The index provides semantic cues for locating evidence, while the original observations retain the details needed for answering. On completed indexes across four video QA benchmarks, **LoreMem** achieves 52.0% average accuracy, exceeding Qwen3-VL-4B caption memory by 7.9 percentage points. Separately, workload evaluation using construction times directly measured on a Galaxy S26 Ultra shows no backlog accumulation over six hours of 30-second arrivals.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
memory construction, online video stream, ordinal lexicon

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_9d8GOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses online memory construction for continuous video streams, aiming to make historical observations searchable while keeping pace with incoming video. It proposes LoreMem, a training-free method that maps visual observations and dialogue to a fixed lexicon and stores sparse concept identities and ranks. At query time, these ranks are used to retrieve original video segments for answering. Experiments evaluate retrieval and QA quality on four long-form video benchmarks, alongside construction workload using measured mobile-device processing times.

Critical strengths

S1: The proposed method effectively addresses the need to construct useful, searchable memories as video continues to arrive.

Its lightweight indexing supports retrieval over extended video histories, improving QA accuracy across four long-form video benchmarks. Mobile-device measurements further support its ability to keep pace with incoming segments without accumulating construction backlog.

Critical weaknesses

W1: The comparisons lack representative streaming models and memory mechanisms relevant to the paper’s motivation.

Although the evaluated benchmarks assess long-term video memory, the baselines mainly cover direct QA, caption memory, and RAG systems. Comparisons omit methods such as Flash-VStream, Dispider, Memento, TimeChat-Online, StreamForest, VST, and JoyAI-VL-Interaction, which offer alternative ways to maintain and use historical information as video arrives. Comparing a representative subset under fair conditions is necessary to clarify LoreMem’s advantages over existing online memory approaches.

W2: This method is limited in its ability to model temporal and contextual relationships.

Each segment is indexed using only its own content, without explicitly linking preceding events. In my opinoin, expressions such as “there are still eight left” and “the three-eyed guy” in Figure 10 highly depend on context, once losing their referents, it causes retrieval failure. Even Figure 9 may become challenging if asked “What did I eat for takeout last night?”: similar meal segments may require surrounding nighttime/daylight cues to distinguish them, but those contextual segments may not be retrieved. Missing context at retrieval cannot be recovered during answering.

References:

[R1] Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams

[R2] Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction

[R3] Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video

[R4] TimeChat-Online: 80% Visual Tokens Are Naturally Redundant in Streaming Videos

[R5] StreamForest: Efficient Online Video Understanding with Persistent Event Memory

[R6] Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

[R7] JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Questions for authors

Q1: Does the proposed method consistently select relevant historical evidence?

Figures 7–10 show only the highest-ranked concepts in ground-truth evidence segments. Could the authors visualize the complete retrieval process, including the concepts selected by the query, their ranks in candidate segments, the final segment scores, and the actual retrieved segments and frames?

Q2: Do high-frequency, generic concepts dominate retrieval scores, making it difficult to distinguish similar segments from different events or causing certain segments to be retrieved frequently?

Could the authors list the most frequent concepts in the Top-M indexes and their frequencies, and report how often each segment is retrieved? Examples comparing concept rankings across repeated activities would also help assess whether the representation is sufficiently discriminative.

Q3: Section 5.1 restricts No-Memory baselines to the latest 300 frames, while retrieval methods select 300 frames from the eligible history. Does this strategy restrict the original models’ capabilities?

Could models supporting full-video input or historical compression access the entire history through their native mechanisms, with accuracy, runtime, and memory costs reported together?

Q4: The proposed method relies heavily on the chosen lexicon. How were the 65,383 WordNet entries selected?

Beyond varying the number of retained concepts, have the authors evaluated whether the underlying lexicon sufficiently covers their video content?

15 comments
AuthorsOpenReview ↗

Author's Response to Reviewer 9d8G (1/6)

We sincerely thank the reviewer 9d8G for their constructive feedback and recognition of the effectiveness of our proposed method in constructing searchable memories from continuously arriving video streams, its improved QA performance across four long-form video benchmarks, and its practical feasibility demonstrated by lightweight indexing and backlog-free operation on a mobile device. We address the reviewer's concerns in detail below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

W1: The comparisons lack representative streaming models and memory mechanisms relevant to the paper’s motivation.

Although the evaluated benchmarks assess long-term video memory, the baselines mainly cover direct QA, caption memory, and RAG systems. Comparisons omit methods such as Flash-VStream, Dispider, Memento, TimeChat-Online, StreamForest, VST, and JoyAI-VL-Interaction, which offer alternative ways to maintain and use historical information as video arrives. Comparing a representative subset under fair conditions is necessary to clarify LoreMem’s advantages over existing online memory approaches.

Thank you for suggesting comparisons with streaming video-memory methods. We have evaluated all seven methods listed in the review: Flash-VStream, StreamForest, Dispider, VST, JoyAI-VL-Interaction (JoyAI), TimeChat-Online, and Memento. Their results are included in Appendix D.1 (Table 8), and their memory mechanisms are discussed in Section 2 of the revised manuscript.

The evaluation uses the 500-question EgoLifeQA subset reported in Appendix A.1 (Table 5). All methods are evaluated under the same input conditions, receiving video frames and aligned dialogue transcripts together with the same textual questions. Each system uses its released memory mechanism and corresponding answer model, with the Memento answerer replacement marked by †\dagger in the table.

LoreMem achieves 49.4%49.4\% QA accuracy, exceeding all seven streaming baselines by 5.25.2–13.413.4 percentage points. Dispider obtains the strongest baseline result at 44.2%44.2\%, followed by TimeChat-Online at 43.4%43.4\% and Memento at 43.0%43.0\%. LoreMem's gains over these methods are 5.25.2, 6.06.0, and 6.46.4 percentage points, respectively.

MethodQA (%)
LoreMem49.4
Flash-VStream36.0
StreamForest41.2
Dispider44.2
VST40.2
JoyAI40.4
TimeChat-Online43.4
Memento †\dagger43.0

†\dagger marks the Memento answerer replacement with Qwen3-VL-8B, as specified in the caption of Table 8.

This comparison evaluates complete memory systems under shared input conditions, using their respective answer models. It complements the fixed-answerer history-selection control in Appendix D.1 (Table 7), where LoreMem and uniform sampling both use Qwen3-VL-8B and a 300-frame answer budget. LoreMem achieves 52.0%52.0\% average QA accuracy compared with 41.3%41.3\% for uniform sampling, a gain of 10.710.7 percentage points. Together, these comparisons support the utility of LoreMem's historical evidence selection under the evaluated settings.

AuthorsOpenReview ↗

Author's Response to Reviewer 9d8G (2/6)

W2: This method is limited in its ability to model temporal and contextual relationships.

Each segment is indexed using only its own content, without explicitly linking preceding events. In my opinoin, expressions such as “there are still eight left” and “the three-eyed guy” in Figure 10 highly depend on context, once losing their referents, it causes retrieval failure. Even Figure 9 may become challenging if asked “What did I eat for takeout last night?”: similar meal segments may require surrounding nighttime/daylight cues to distinguish them, but those contextual segments may not be retrieved. Missing context at retrieval cannot be recovered during answering.

Thank you for highlighting these context-dependent cases. LoreMem retains temporal metadata and pointers to the original video and dialogue alongside each segment's semantic index. The stored concept ranks provide retrieval cues, while the original observations remain available for contextual inspection. The context preserved in memory and the context selected by retrieval are therefore distinct.

For time-qualified questions such as “last night,” segment timestamps and the question time provide explicit temporal information. The EgoLifeQA/Ego-R1 answer prompt includes both fields (Appendix F.1, Table 18). A retrieval policy could use this information to constrain the relevant time window, without depending exclusively on surrounding nighttime/daylight frames. Retaining this information alone does not ensure that the required segment is selected.

Alongside expressions such as “there are still eight left” and “the three-eyed guy,” a segment's visual and dialogue memories provide additional semantic cues. LoreMem aggregates the stored ranks of N=100N=100 query-selected concepts across both modalities, allowing other cues to support retrieval when a particular expression has an unclear referent. Concepts with identical lookup ranks across candidates contribute equally to their costs, while differences in other concept ranks provide discrimination. Figure 5 shows that increasing query support from N=10N=10 to N=100N=100 improves nDCG@20 on all four benchmarks, supporting the use of multiple query concepts. Explicit cross-segment reference resolution remains a limitation of the current single-pass protocol. Figure 10 illustrates a mismatch between the return-related query concepts and the annotated claw-machine event; it does not isolate the effect of cross-event relationships.

An agentic retrieval policy could inspect neighboring segments or issue follow-up searches over the preserved observations before answering. Such policies could retain LoreMem's segment-local construction procedure while adding query-time computation. This separation is consistent with our focus on building reusable indexes before future questions are known and keeping construction workload bounded under sustained arrivals. Appendix E of the revised manuscript now states the absence of explicit cross-segment relationships and discusses these complementary query-time mechanisms as possible extensions.

AuthorsOpenReview ↗

Author's Response to Reviewer 9d8G (3/6)

Q1: Does the proposed method consistently select relevant historical evidence?

Figures 7–10 show only the highest-ranked concepts in ground-truth evidence segments. Could the authors visualize the complete retrieval process, including the concepts selected by the query, their ranks in candidate segments, the final segment scores, and the actual retrieved segments and frames?

Thank you for requesting a trace from the query to the retrieved evidence. We have expanded Appendix D.7 and Figures 7–10 to show query-concept selection, candidate memory ranks and costs, final retrieval positions, and corresponding original observations. Panel (a) displays six representative concepts from the query's top-N=100N=100 entries, with their query ranks and cosine similarities. Panel (b) reports candidate timestamps, visual and dialogue lookup ranks, costs, and retrieval positions; panel (c) provides frames and transcript excerpts. Figures 7–8 compare the top three candidates, while Figures 9–10 compare the top two candidates with the ground-truth segment, including its position when it falls outside the retrieved top 20.

The figures follow Section 4.4: concepts outside the retained top-M=1,000M=1{,}000 memory receive the lookup rank M+1=1,001M+1=1{,}001, and retrieval selects the K=20K=20 segments with the lowest fused cost, Dtxt+0.1DvisD_{\mathrm{txt}}+0.1D_{\mathrm{vis}}. The Cost (6) columns and their fused values are subtotals over the six illustrated concepts; the Cost (100) columns and final retrieval positions use all 100 query concepts. Appendix D.7 makes this distinction explicit, since the six-concept subtotals need not reproduce the full retrieval order.

The successful examples connect the retrieved segments to specific factual evidence. In Figure 7, the takeout segment is retrieved first: takeout ranks third in visual memory and second in dialogue memory, while the original dialogue identifies KFC roast duck and KFC pizza. Food-related alternatives are shown alongside it, allowing their concept support to be compared. In Figure 8, the coin-price segment is also retrieved first. Its dialogue memory ranks coin first and ego third, whereas the other displayed monetary discussions do not retain ego. The retrieved transcript supplies the precise answer, “each coin is now 3 RMB.”

Figures 9–10 expose different retrieval outcomes. In Figure 9, the annotated arcade segment ranks 9 and remains within the top-20 evidence set. Its visual memory ranks arcade, playing, and game highly, while its dialogue memory matches few of the displayed query concepts. In Figure 10, return-related query concepts match unrelated returning scenes; the annotated claw-machine segment ranks 876 and is excluded. This failure trace makes the mismatch between query-selected concepts and relevant evidence visible, alongside the successful cases.

These examples connect concept-level indexing to actual evidence inclusion or exclusion. Across benchmarks, Table 6 provides aggregate retrieval evidence: LoreMem achieves average temporal-evidence nDCG@20 of 18.8, compared with 7.2 for the strongest visual-embedding baseline and 9.7 for local caption memory. The expanded traces explain concrete retrieval outcomes and complement this measured retrieval utility.

Reviewer_YHfzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that video-memory research has optimized retrieval quality while largely neglecting whether memory construction can keep pace with an incoming stream, and formulates online memory construction as a joint quality–stability problem: the index must support useful retrieval while keeping outstanding construction work bounded under sustained arrivals and a fixed compute budget. Stability is measured through a queue recursion (Eqs. 1–2) whose normalized workload ρ_t = V_t/Δ accounts for backlog carried over from earlier segments rather than per-segment construction time in isolation. The proposed method, LoreMem, builds on the observation that linguistic indexing does not require generating language: rather than captioning each segment, it projects frames and dialogue onto a fixed 65,383-entry WordNet lexicon using two frozen encoders (PE-Core-B16 for vision, BGE-Large or BGE-M3 for text) and stores only the top-M = 1,000 concept identities and their ranks per modality per 30-second segment. A query is projected into the same lexicon, and retrieval sums the stored ranks of the query's top-N concepts, fusing modalities through a single visual weight λ — an ordinal formulation that deliberately sidesteps calibrating raw similarity scales across two encoder spaces. The retained original observations, not the index, supply the detail used for answering. Across four long-form egocentric benchmarks LoreMem reaches 52.0% average QA accuracy and 18.8 average nDCG@20, with the clearest margins on Ego-R1 and SuperMemory-VQA, while directly measured on-device construction times on a Galaxy S26 Ultra show no backlog accumulation over six hours of 30-second arrivals, in contrast to the caption builder. The paper is clearly written, the formulation is well motivated, and using a shared lexicon to make two independent frozen encoders interoperate is an elegant design. My reservations are about cost accounting and robustness: the storage footprint of retaining every frame alongside a fairly dense per-segment index is never quantified, the global frame-level representation provides no mechanism for locating small-scale targets, and the TeleEgo results are flat in a way the paper interprets favorably rather than diagnoses.

Critical strengths

Strengths

  1. The paper is well written and well motivated.
  2. I really liked the idea of using separate vision and text encoders to build a memory bank. 
  3. Results are also looking good on Ego-R1 and SuperMemory-VQA.
Critical weaknesses

Weakness 

  1. The computational cost of storing every frame seems too high. Is it possible to even abstract out the details or store lower dimension for every frame and upsample whenever more dense information is needed?

  2. How to handle cases where scale changes, for ex. If a person is looking for a small thing in the frame whose scale might be small, how will the model find that? Ideally that will require memory to be able to look up past frames and zoom in or zoom out.

  3. Why are the results mixed on TeleEgo? Improvement over caption style is pretty low.

Questions for authors

There are some small concerns in results specially on TeleEgo dataset. Other than that analysis of the paper is also weak, I'm not really sure what is being learnt in WordNet lexicon and linguistic indexing .

6 comments
AuthorsOpenReview ↗

Author's Response to Reviewer YHfz (1/3)

We sincerely thank the reviewer YHfz for their constructive feedback and positive assessment of the clarity and motivation of our work, the design of our memory bank using separate vision and text encoders, and the promising results on Ego-R1 and SuperMemory-VQA. We address the reviewer's concerns in detail below.

revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

[W1] Sparse index cost and original-evidence storage The computational cost of storing every frame seems too high. Is it possible to even abstract out the details or store lower dimension for every frame and upsample whenever more dense information is needed?

We appreciate the reviewer's intuition that memory should provide a compact abstraction while preserving access to finer details when needed. LoreMem already abstracts the sampled observations into a segment-level semantic index. For each 30-second segment, the evaluated protocol encodes 15 frames at 0.5 fps and pools their concept affinities into one visual record; dialogue produces one text record. Each record retains only the top-M=1,000 concept identities and ranks, approximately 1.53% of the 65,383-entry lexicon. The full visual/text anchor tables are stored once and shared across segments. Frame embeddings are used during construction and need not be retained in the searchable index.

An optimized encoding can store each modality's concept IDs in rank order using uint16. The array position gives the rank, so a separate rank array is unnecessary; concepts outside the stored support receive the lookup rank M+1, as in Section 4.4. This preserves the ordinal record exactly and requires 2 modalities × 1,000 IDs × 2 bytes = 4,000 bytes per segment. The resulting analytical payload estimates are:

Recording durationSegmentsOptimized index payload
1 hour1200.46 MiB
6 hours7202.75 MiB
24 hours2,88010.99 MiB

These estimates exclude metadata, serialization overhead, retained images/text, and model/runtime memory. The shared anchors add a fixed cost of L(dvis+dtxt)bL(d_{vis}+d_{txt})b bytes. For 1,024-dimensional visual and text anchors, FP16 storage would require approximately 255.4 MiB, independently of recording duration. Both calculations describe assumed storage formats, rather than measurements of the current implementation. The uint16 ID encoding preserves the index exactly; the FP16 anchor estimate does not establish unchanged rankings after reducing floating-point precision.

The existing support ablation also evaluates this abstraction: Figure 5 shows that increasing M from 10 to 1,000 improves nDCG@20 on all four benchmarks, while retaining the entire lexicon brings benchmark-dependent changes. Thus, the sparse support is evaluated for retrieval utility as well as its representational size.

Our current implementation separately stores frame images for answer-time evidence. All retrieval methods in the shared-answerer comparison access original frames and transcripts, with a fixed 300-frame answer budget (Section 5.1), so access to source evidence is a common requirement of that evaluation. Its storage cost remains additional to the semantic index. The index can locate a conversation through coin-related concepts, while the transcript preserves the exact value of 3 RMB (Figure 8); similarly, frames preserve visual details for answering. Keeping only low-dimensional features or reduced-resolution images and upsampling them later would not guarantee recovery of those details. Compressed source storage with access restricted to retrieved intervals is a possible deployment option; we do not evaluate it here. Appendix F.1 (Tables 18 and 19) specifies the original observations used for answering; the analytical payload estimates above concern only the semantic index.

AuthorsOpenReview ↗

Author's Response to Reviewer YHfz (2/3)

[W2] Small objects, scale changes, and spatial inspection How to handle cases where scale changes, for ex. If a person is looking for a small thing in the frame whose scale might be small, how will the model find that? Ideally that will require memory to be able to look up past frames and zoom in or zoom out.

Thank you for highlighting the challenge of retrieving small objects across changes in scale. Small-object and scale sensitivity depends on the visual information captured by the frozen PE-Core encoder. LoreMem constructs a reusable semantic memory from these features before future queries are known. Within each segment, concept-wise max pooling over 15 sampled frames allows a cue strongly represented in any sampled view to contribute to the memory, including when other views are less informative.

Retrieval combines multiple query-selected concepts across visual and dialogue memories. Contextual scene, activity, or dialogue cues can therefore help locate a segment even when the target object's own cue is weak; the retrieved original frames and transcripts supply the details for answering. Table 14 supports the utility of this visual branch: on SuperMemory-VQA, adding vision raises nDCG@20 from 15.2 for dialogue-only LoreMem to 21.5 (+6.3+6.3 points), and Figure 6 shows improved object-location and visual-recall retrieval as visual weight increases. These are temporal-retrieval results; they do not separately establish small-object or scale robustness.

Temporal max pooling provides no spatial zoom. Retrieval can still fail if a target is absent from the sampled frames, weakly encoded, or omitted from the top-M support. Retaining source images permits inspection of retrieved evidence, but cannot repair a missed segment. Adaptive localization and zoom would require additional perception and evidence-inspection capabilities beyond the evaluated protocol. Appendix E now explicitly states the dependence on frozen encoder resolution and sampled views; the reported visual-branch results concern temporal evidence retrieval.


[W3] TeleEgo's mixed results and their interpretation Why are the results mixed on TeleEgo? Improvement over caption style is pretty low.

We agree that the completed-index results show limited QA improvement over caption memory on TeleEgo: both achieve 54.7%, with caption memory slightly higher in nDCG@20 (15.5 versus 15.0; Tables 1 and 6). However, the central problem studied here is whether useful memory can be constructed fast enough to keep pace with incoming video.

Our reported on-device builder profiling gives 0.06 seconds per 30-second segment for LoreMem versus 37.84 seconds for the default Qwen3-VL-4B caption memory with a 1,024-token cap (Appendix D.2, Table 9). In a continuing stream, this slower construction can leave arrived observations unindexed when a query arrives. Completed-index QA, as used in the original submission, assumes away this availability gap, granting caption memory access to observations that may still be unindexed online.

Because the builders differ substantially in construction latency, evaluating QA together with construction progress is more appropriate for assessing their practical streaming utility. We therefore evaluate both methods at the same query timestamps, tracking memory construction and the latency of answering earlier queries. Retrieval admits only temporally eligible records whose construction has finished by query time. Both methods use the same Qwen3-VL-8B answerer and a 300-frame budget. Under this evaluation, LoreMem retains its 54.7% accuracy on TeleEgo, while caption memory falls to 53.8% (Section 5.2, Table 2).

This small aggregate gap partly reflects the answerer itself: the same answerer achieves 51.1% accuracy from the question and options alone, without video or dialogue evidence. This limits the diagnostic value of aggregate QA for evaluating retrieval-based memory [1, 2]. We also report QA on the 852 questions this baseline answers incorrectly:

Memory methodAll questions (N = 1,743)Question-only errors (N = 852)
LoreMem (online)54.7%29.81%
Qwen3-VL-4B caption (completed index)54.7%29.11%
Qwen3-VL-4B caption (online)53.8%24.30%

On these questions, online availability reduces caption accuracy by 4.81 percentage points, while LoreMem exceeds online caption memory by 5.51 points.

[1] Balepur, Nishant, Abhilasha Ravichander, and Rachel Rudinger. "Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?" ACL 2024.

[2] Chen, Lin, et al. "Are We on the Right Way for Evaluating Large Vision-Language Models?" NeurIPS 2024.

AuthorsOpenReview ↗

Author's Response to Reviewer YHfz (3/3)

[Q1] There are some small concerns in results specially on TeleEgo dataset. Other than that analysis of the paper is also weak, I'm not really sure what is being learnt in WordNet lexicon and linguistic indexing.

Thank you for this feedback. Beyond the analyses in the original submission (pooling/scoring, segment duration, memory/query support, modality balance, and construction workload; Sections 5.2–5.3), the revision adds direct retrieval and lexicon controls (Appendices D.6 and D.3), expanded retrieval traces (Appendix D.7), and concept-frequency and retrieval-concentration analyses (Appendix D.8). We additionally report the TeleEgo diagnostics above. The following analyses clarify what the linguistic index captures and how that information supports retrieval.

LoreMem learns no new model parameters from WordNet; it records concept support and ranks for each video segment. The fixed lexicon provides shared concept identities, and pretrained, frozen encoders determine their support in the segment's frames and dialogue. Queries select concepts from the same inventory, and their stored ranks contribute to candidate scores across both modalities. The index supplies retrieval cues; the original frames and transcripts retain the factual details used for answering.

Figure 11 examines three poker episodes with similar frames but different conversations: increasing the blinds, breakfast, and ordering food. The visual concept poker ranks 10, 6, and 6 across the episodes, providing little separation. Query-selected dialogue concepts increase, breakfast, and ordering instead have stronger retained support in their corresponding episodes. Using the full 100-concept retrieval score, the matching episodes rank 1, 2, and 1 among 376 eligible candidates. This illustrates how the index captures distinctions between episodes sharing the same visual activity.

We also test whether frequently retained concepts dominate retrieval. On annotated SuperMemory-VQA questions, removing scoring contributions from the 50 or 500 most frequent concepts per source in each eligible candidate pool, with query concepts and candidate pools fixed, gives nDCG@20 of 21.5 and 21.4, versus 21.5 without removal (Appendix D.8, Table 16). The tested frequent-concept groups have little net effect on aggregate retrieval quality in this setting; ambiguity between repeated events remains a limitation.

Quantitative controls further support the representation's utility: full LoreMem raises mean nDCG@20 from 14.3 to 18.8 over lexicon-free BGE/PE candidate-rank fusion (Appendix D.6, Table 14). Within the fixed lexicon, replacing raw affinity sums with ordinal scoring, while keeping max pooling, raises EgoLifeQA accuracy from 41.6% to 49.4% (Table 4).

Reviewer_nohsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies online memory construction for long-form video assistants: making incoming segments searchable while keeping up with the stream under a fixed compute budget. The authors frame this as a joint quality-stability problem and measure stability as the normalized backlog of a single first-in, first-out builder (Eq. 1–2). They propose LoreMem, a non-parametric, caption-free memory. Frozen visual and text encoders project each segment’s frames and dialogue onto a fixed lexicon of 65,383 WordNet entries, and the memory stores the top-M entries and their ranks per modality. Retrieval sums the stored ranks of the lexicon entries the query activates, with a weighted visual term. On four egocentric benchmarks, LoreMem reaches 52.0% average QA accuracy, 7.9 points above a Qwen3-VL-4B caption memory and 4.1 above Video-RAG. Construction times measured on a Galaxy S26 Ultra show no backlog over six hours of 30-second segments.

Critical strengths
  • The problem is relevant. Whether memory construction keeps pace with the stream under a mobile budget is a real constraint for wearable assistants.
  • The method is simple, needs no training, and works with any frozen encoder. Using ranks avoids calibrating similarity scales across encoders, and Table 3 shows it matters (41.6 -> 49.4).
  • Gains are consistent across benchmarks for both QA and retrieval (average nDCG@20 of 18.8 vs. 9.7 for caption memory).
  • Construction times are measured on a real device, and the qualitative examples, including a failure case, make the representation easy to understand.
Critical weaknesses
  • The stability claim rests on narrow evidence. Only one caption builder is timed on the device: Qwen3-VL-4B with Q4_0 quantization and a 1024-token cap. Its mean of about 40 s against a 30 s interval (Fig. 1) means a builder only about 25% faster could keep up. Newer and faster options are not evaluated: smaller VLMs (e.g., Qwen3-VL-2B), hybrid linear-attention VLMs (e.g., the Qwen3.5 small models with Gated DeltaNet or more recent versions), and faster decoding through speculative decoding (Gagrani et al., 2024) or multi-token prediction. In addition, the 512-token variant is never timed (App. D.3) and Video-RAG, EgoRAG and LightRAG are not timed as well.

  • The size of the caption comparison may depend on the choice of captioner. The on-device Qwen3-VL-4B caption memory retrieves relatively weakly on EgoLifeQA and Ego-R1: N@20 of 3.7 and 2.3, below every visual embedding on EgoLifeQA and on par with them on Ego-R1 (Table 5). With GPT-5.2 captions, caption memory reaches 50.9 average accuracy versus LoreMem’s 52.0, with slightly higher average N@20 (19.8 vs. 18.8; Table 6). GPT-5.2 is a cloud model and not a deployment alternative, so this does not undermine the on-device comparison. It does suggest that part of the 7.9-point gap may reflect the particular captioner rather than captioning as such. It would help to know where the quality-cost trade-off lies for on-device captioners.

  • LoreMem combines projecting onto the lexicon with scoring by rank. On EgoLifeQA, the lexicon alone hurts (41.6, Table 3, vs. 47.2 for raw fusion without it, Table 7), and ranking recovers it (49.4). Ranking without the lexicon is never tested, so it is unknown whether the lexicon adds anything; the authors acknowledge this (l.753–754). LoreMem at λ = 0 is also not reported, even though retrieval is mostly driven by dialogue and direct dialogue retrieval already reaches 48.0. Gaps this small would benefit from significance testing.

  • Generality beyond the tested setting is not studied. The lexicon is fixed to English WordNet (65,383 entries), with no description of how the entries were selected and no ablation of its size or content. Missing concepts (e.g., the claw machine in Fig. 10), the mapping of Chinese TeleEgo data onto the English lexicon, and robustness to speech recognition errors (all methods receive clean transcripts) are not analyzed.

Questions for authors
  • Can you time more on-device builders and report the results as numbers? For example, the Qwen3.5 small models, the 512-token cap, and the Video-RAG, EgoRAG and LightRAG builders.
  • For on-device captioners, how does accuracy and N@20 trade off against construction time across model sizes?
  • On all four benchmarks, how does LoreMem at λ = 0 compare with rank-based fusion of BGE and Perception Encoder scores without the lexicon? Are the gaps significant under a paired test?
  • How do results change with lexicon size and content?
8 comments
AuthorsOpenReview ↗

Author's Response to Reviewer nohs (1/4)

We sincerely thank the reviewer nohs for their constructive feedback and recognition of the practical importance of online memory construction under mobile resource constraints, the simplicity and training-free nature of our encoder-agnostic method, and its consistent improvements in QA and retrieval performance across benchmarks. We also appreciate the reviewer's positive assessment of our real-device measurements and qualitative analyses. We address the reviewer's concerns in detail below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

[W1, Q1, Q2] Breadth of the on-device construction and stability comparison The stability claim rests on narrow evidence. Only one caption builder is timed on the device: Qwen3-VL-4B with Q4_0 quantization and a 1024-token cap. Its mean of about 40 s against a 30 s interval (Fig. 1) means a builder only about 25% faster could keep up. Newer and faster options are not evaluated: smaller VLMs (e.g., Qwen3-VL-2B), hybrid linear-attention VLMs (e.g., the Qwen3.5 small models with Gated DeltaNet or more recent versions), and faster decoding through speculative decoding (Gagrani et al., 2024) or multi-token prediction. In addition, the 512-token variant is never timed (App. D.3) and Video-RAG, EgoRAG and LightRAG are not timed as well. Can you time more on-device builders and report the results as numbers? For example, the Qwen3.5 small models, the 512-token cap, and the Video-RAG, EgoRAG and LightRAG builders. For on-device captioners, how does accuracy and N@20 trade off against construction time across model sizes?

Thank you for requesting a broader comparison of on-device builders. We have added the 512-token Qwen3-VL-4B configuration, Qwen3-VL-2B, and Qwen3.5-0.8B in Appendix D.2 (Table 9), and report QA accuracy, retrieval quality, and construction time on EgoLifeQA together below. Construction time is reported per 30-second segment; N@20 denotes temporal-evidence nDCG@20 on the paper's reported scale.

Construction configurationCaption output capQA (%)N@20Construction Time (s)
LoreMemNot applicable49.415.10.06
Qwen3-VL-4B1,02440.63.737.84
Qwen3-VL-4B51235.03.627.95
Qwen3-VL-2B51234.83.420.96
Qwen3.5-0.8B51234.63.413.51

The added results support the reviewer's point that faster caption configurations can reduce construction time below the 30-second arrival interval. Reducing the 4B output cap from 1,024 to 512 tokens lowers construction time from 37.84 to 27.95 seconds, a 26.1% reduction, but also lowers QA accuracy from 40.6% to 35.0%, with N@20 changing from 3.7 to 3.6. Qwen3.5-0.8B further reduces construction time to 13.51 seconds, while achieving 34.6% QA and 3.4 N@20. Thus, the added comparison includes faster caption configurations and makes their observed quality–cost trade-off explicit.

LoreMem achieves 49.4% QA and 15.1 N@20 with a construction time of 0.06 seconds. Compared with Qwen3.5-0.8B, the fastest measured caption configuration in this table, its QA is 14.8 percentage points higher and N@20 is 11.7 points higher, while its reported construction time is approximately 225 times shorter. It also exceeds the original 4B/1,024-token configuration in both quality metrics. These results show that LoreMem's advantage persists when the comparison includes caption builders whose reported construction times fall below the arrival interval.

The six-hour workload result in Figure 3 uses measured service times for LoreMem and the original 4B/1,024-token builder. The added per-segment timings broaden the quality–cost comparison; long-duration workload trajectories for those configurations are not reported here. Matched on-device timings for Video-RAG, EgoRAG, LightRAG, and speculative or multi-token decoding variants are also outside the added measurements.

Appendix D.2 now reports these quality–cost results. Section 5.2 and Appendix C describe the separate six-hour workload evaluation using the default caption configuration. Our contribution is efficient construction of a useful, reusable memory: across the tested local caption configurations, LoreMem provides higher QA and retrieval quality with substantially lower construction cost.

AuthorsOpenReview ↗

Author's Response to Reviewer nohs (2/4)

[W2] Dependence on captioner quality The size of the caption comparison may depend on the choice of captioner. The on-device Qwen3-VL-4B caption memory retrieves relatively weakly on EgoLifeQA and Ego-R1: N@20 of 3.7 and 2.3, below every visual embedding on EgoLifeQA and on par with them on Ego-R1 (Table 5). With GPT-5.2 captions, caption memory reaches 50.9 average accuracy versus LoreMem’s 52.0, with slightly higher average N@20 (19.8 vs. 18.8; Table 6). GPT-5.2 is a cloud model and not a deployment alternative, so this does not undermine the on-device comparison. It does suggest that part of the 7.9-point gap may reflect the particular captioner rather than captioning as such. It would help to know where the quality-cost trade-off lies for on-device captioners.

We agree that captioner quality affects memory performance. The 7.9-percentage-point average QA gain is relative to the evaluated local Qwen3-VL-4B caption memory. We included GPT-5.2 as a stronger caption-quality reference: it achieves 50.9% average QA and 19.8 average nDCG@20, compared with LoreMem's 52.0% and 18.8. Its higher retrieval score demonstrates the benefit of strong captions, while LoreMem provides competitive QA and retrieval utility without autoregressive caption generation during memory construction.

This stronger caption memory also has a measurable construction cost. Captioning the seven days of EgoLifeQA video clips with GPT-5.2 incurred approximately USD 64 in API charges in our experiment. Additional incoming video requires further caption generation and associated API expenditure. LoreMem constructs its index locally using frozen encoders, avoiding cloud caption-generation charges while retaining the original observations for answering.

The added local results in the preceding response further characterize the quality–cost trade-off. Qwen3.5-0.8B reduces construction time to 13.51 seconds per 30-second segment, below the arrival interval, but achieves 34.6% EgoLifeQA QA and 3.4 N@20. LoreMem achieves 49.4% QA and 15.1 N@20 in 0.06 seconds per segment. The 4B/512-token and 2B/512-token configurations likewise achieve lower QA and retrieval scores. Thus, the observed benefit extends across the tested local caption configurations, including faster builders.

These comparisons support LoreMem's construction objective: useful, reusable memory can be built with QA and retrieval utility close to the strong cloud-caption reference, while achieving higher quality and substantially lower construction time than the tested local caption builders. The revised manuscript includes the strong-caption quality comparison in Appendix D.4 (Table 12) and the additional local quality–cost results in Appendix D.2 (Table 9). The measured GPT-5.2 caption API expenditure is reported above.

AuthorsOpenReview ↗

Author's Response to Reviewer nohs (3/4)

[W3, Q3] Dialogue strength, visual contribution, and lexicon/rank attribution LoreMem combines projecting onto the lexicon with scoring by rank. On EgoLifeQA, the lexicon alone hurts (41.6, Table 3, vs. 47.2 for raw fusion without it, Table 7), and ranking recovers it (49.4). Ranking without the lexicon is never tested, so it is unknown whether the lexicon adds anything; the authors acknowledge this (l.753–754). LoreMem at λ=0\lambda=0 is also not reported, even though retrieval is mostly driven by dialogue and direct dialogue retrieval already reaches 48.0. Gaps this small would benefit from significance testing. On all four benchmarks, how does LoreMem at λ=0\lambda=0 compare with rank-based fusion of BGE and Perception Encoder scores without the lexicon? Are the gaps significant under a paired test?

Thank you for asking us to clarify the contributions of dialogue, vision, and lexicon-based indexing. We have added the lexicon-free BGE/Perception Encoder rank-fusion baseline and LoreMem at λ=0\lambda=0 to Appendix D.6 (Table 14), and report paired retrieval tests in this response. Direct rank fusion combines candidate-segment rankings from vision and dialogue (Appendix D.6, Equation 10), whereas LoreMem sums within-segment ranks of query-selected lexicon concepts (Equations 7–8). The table reports temporal-evidence nDCG@20 on the paper's scale; Average is the unweighted benchmark mean. Scores, means, and differences are rounded separately to one decimal place.

Retrieval representationEgoLifeQAEgo-R1SuperMemory-VQATeleEgoAverage
Direct dialogue retrieval11.316.27.712.511.9
LoreMem, λ=0\lambda=016.022.515.213.516.8
Direct BGE/PE rank fusion, λ=0.1\lambda=0.113.617.716.19.714.3
LoreMem, λ=0.1\lambda=0.115.123.521.515.018.8

Full LoreMem improves mean nDCG@20 from 14.3 to 18.8 (+4.5+4.5 points) over direct rank fusion, with gains on every benchmark. We test this comparison with both methods using vision and dialogue, on the same gold-annotated single-MC questions within each benchmark. The two-sided paired randomization results are:

BenchmarkΔnDCG@20\Delta\mathrm{nDCG@20}Paired pp
EgoLifeQA+1.5+1.50.04650.0465
Ego-R1+5.9+5.90.00340.0034
SuperMemory-VQA+5.4+5.40.03560.0356
TeleEgo+5.3+5.35.0×10−65.0\times10^{-6}

Here, Δ\Delta denotes full LoreMem minus direct rank fusion. Each row reports a separate paired test for that benchmark.

For each benchmark, we compute paired question-level score differences and use their absolute mean as the test statistic. In each of B=200,000B=200{,}000 randomizations, we independently flip each difference's sign with probability 1/21/2:

di=nDCG@20iLoreMem−nDCG@20irank fusiond_i = \mathrm{nDCG@20}_i^{\mathrm{LoreMem}} - \mathrm{nDCG@20}_i^{\mathrm{rank\ fusion}}

\quad T_b = \left| \frac{1}{n} \sum_{i=1}^{n} s_i^{(b)} d_i \right|$$ $$p = \frac{1 + \sum_{b=1}^{B} \mathbf{1}(T_b \ge T_{\mathrm{obs}})}{B+1}$$ Here, $n$ is the number of paired annotated questions and $s_i^{(b)}\in\{-1,+1\}$. The absolute statistic gives a two-sided test, and the formula includes the Monte Carlo add-one correction. To assess dialogue indexing without adding vision, **the appropriate control is LoreMem at $\lambda=0$ versus direct dialogue retrieval**. Its mean nDCG@20 rises from 11.9 to 16.8 ($+4.9$ points), with gains on all four benchmarks. For the reviewer's requested comparison with multimodal rank fusion, $\lambda=0$ LoreMem is higher on EgoLifeQA ($+2.4$), Ego-R1 ($+4.8$), and TeleEgo ($+3.8$), but lower on SuperMemory-VQA ($-0.9$); its mean is higher by $2.5$ points. This comparison also changes the available modalities, so it does not isolate the lexicon's contribution. Adding vision within LoreMem increases the mean by $2.0$ points, with its largest gain on SuperMemory-VQA ($+6.3$), while decreasing EgoLifeQA retrieval by $0.9$ points. The earlier 41.6% versus 47.2% QA comparison changes both representation and scoring, rather than lexicon projection alone. Within the lexicon representation, replacing raw concept-affinity sums with ordinal costs raises QA from 41.6% to 49.4% (Table 4). The new rank-fusion comparison supports the utility of the joint lexicon-ordinal design and is included in Appendix D.6 (Table 14). Paired inference reported in this response concerns full LoreMem versus direct rank fusion in retrieval; the $\lambda=0$ and QA differences are descriptive.
Reviewer_AR2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Important topic, promising demo. But lacks enought insights and experiments on the online memory construction. The QA experiments are based on fully constructed memory indexes. See weakness 1 for more details.

If the authors provide some solid experiments and insights on weakness 1, it think this paper can be accepted.

Critical strengths
  1. The method is simple yet effective for online memory construction.
  2. The authors implement their system on a Galaxy S26 Ultra shows no backlog accumulation over six hours of 30-second arrivals. This is a very sold result.
Critical weaknesses
  1. Lack of end-to-end online evaluation. QA is evaluated on fully constructed memory indexes, while construction latency is assessed separately. The paper does not evaluate retrieval and QA using only the memory actually available at query time, leaving its effectiveness under realistic streaming conditions unverified.

The implemenation is Galaxy is promising, but it is more like a technical contribution and offer us little insights to the online memory construction problem itself. The retrieval system is so classical that it has been widely adapted. The authors should have dive deeper into the problems in online/streaming settings.

  1. This system has limited low-level visual retrieval capability. LoreMem relies heavily on frozen encoder priors and a fixed lexicon, limiting its ability to retrieve fine-grained visual details (e.g., clothing textures and materials). I do not consider it to be a major weakness since high-level semantic can slove plenty real life problems.
Questions for authors

If the authors provide some solid experiments and insights on weakness 1, it think this paper can be accepted.

5 comments
AuthorsOpenReview ↗

Author's Response to Reviewer AR2p (1/2)

We sincerely thank the reviewer AR2p for their constructive feedback and recognition of the simplicity and effectiveness of our proposed online memory construction method, as well as its practical feasibility demonstrated through sustained on-device operation without backlog accumulation. We address the reviewer's concerns in detail below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

W1: Lack of end-to-end online evaluation QA is evaluated on fully constructed memory indexes, while construction latency is assessed separately. The paper does not evaluate retrieval and QA using only the memory actually available at query time, leaving its effectiveness under realistic streaming conditions unverified.

Thank you for this excellent suggestion, which directly tests the practical value of timely memory construction. We initially used completed indexes to compare retrieval quality over the same temporally eligible history, without additionally penalizing caption memory for construction lag. Following your suggestion, the online evaluation now accounts for both memory construction and the latency of answering earlier queries.

For each query qq with timestamp τq\tau_q, we track memory-construction progress along a timeline that includes captioning the incoming context and answering queries with earlier timestamps. Let FtF_t be the completion time of memory record mtm_t on this timeline. Retrieval uses only

=\left\\{m_t\in\mathcal{C}^{\mathrm{eligible}}_q:F_t\le\tau_q\right\\},$$ where $\mathcal{C}^{\mathrm{eligible}}_q$ contains the records allowed by the benchmark's temporal rules. A segment that arrives before the query remains unavailable until its memory construction finishes; earlier answering work is included when determining this progress. All methods use the same Qwen3-VL-8B answerer and $300$-frame budget. The table reports QA accuracy (%), with an unweighted benchmark mean. "Offline / Online" denotes identical reported accuracies for both evaluations. | Memory method | Evaluation | EgoLifeQA | Ego-R1 | SuperMemory-VQA | TeleEgo | Average | | -------------------------- | ---------------- | --------: | -----: | --------------: | ------: | ------: | | LoreMem | Offline / Online | 49.4 | 56.0 | 48.0 | 54.7 | 52.0 | | OpenCLIP | Offline / Online | 37.6 | 35.0 | 37.1 | 53.6 | 40.8 | | Perception Encoder | Offline / Online | 45.2 | 40.0 | 38.3 | 52.7 | 44.1 | | MobileCLIP | Offline / Online | 42.2 | 32.7 | 37.9 | 52.2 | 41.3 | | SigLIP2 | Offline / Online | 39.4 | 33.7 | 37.9 | 52.8 | 41.0 | | Qwen3-VL-4B Caption memory | Offline | 40.6 | 37.0 | 44.2 | 54.7 | 44.1 | | Qwen3-VL-4B Caption memory | Online | 34.2 | 35.3 | 39.4 | 53.8 | 40.7 | **LoreMem and all four visual-embedding baselines retain their offline QA accuracies under this evaluation schedule.** Caption memory declines on every benchmark: by $6.4$, $1.7$, $4.8$, and $0.9$ percentage points on EgoLifeQA, Ego-R1, SuperMemory-VQA, and TeleEgo, respectively. **Its reported mean QA falls from $44.1\%$ to $40.7\%$ ($-3.4$ percentage points).** The largest drops occur on EgoLifeQA and SuperMemory-VQA, showing that completed-index QA can overestimate caption memory's observed utility when retrieval must use the pool available after both construction and previous answering work. LoreMem maintains $52.0\%$ average QA; the gap between its and caption memory's reported means therefore widens from $7.9$ percentage points offline to $11.3$ online. These results support the practical value of quickly indexing incoming observations while the pipeline also handles queries.
AuthorsOpenReview ↗

Author's Response to Reviewer AR2p (2/2)

W2: The implemenation is Galaxy is promising, but it is more like a technical contribution and offer us little insights to the online memory construction problem itself. The retrieval system is so classical that it has been widely adapted. The authors should have dive deeper into the problems in online/streaming settings.

Thank you for asking us to clarify the insights for online memory construction. Our contribution is fast, query-independent construction of reusable memory on edge devices under sustained video arrivals. Section 3 models accumulated unfinished work: strong completed-index retrieval is insufficient when construction leaves recent evidence unavailable. LoreMem constructs indexes without autoregressive descriptions, and its evaluated six-hour workload shows no backlog under 3030-second arrivals.

Streaming systems such as Flash-VStream and StreamForest compress history under memory budgets, which can discard details needed by future questions. LoreMem retains independent segment records linked to their original frames and dialogue. Appending new segments leaves past observations accessible; successful retrieval can expose details beyond the sparse concept index. Storage grows with history, while selected evidence fits a 300300-frame answer context. This makes retrieve-then-generate a practical design for retrospective QA.

Appendix D.1 (Table 8) compares seven streaming systems under the same video, aligned dialogue, and question inputs, using their respective answer models:

MethodQA (%)
LoreMem49.4
Flash-VStream36.0
StreamForest41.2
Dispider44.2
VST40.2
JoyAI40.4
TimeChat-Online43.4
Memento †\dagger43.0

†\dagger marks the Memento answerer replacement with Qwen3-VL-8B, as specified in the caption of Table 8.

LoreMem achieves 49.4%49.4\% QA, exceeding the strongest streaming baseline by 5.25.2 percentage points. The fixed-answerer control (Table 7) further tests evidence selection: with Qwen3-VL-8B and 300300 frames, LoreMem reaches 52.0%52.0\% average QA versus 41.3%41.3\% for uniform history sampling. These results support selecting relevant historical evidence within a bounded answer context.

We also examine construction trade-offs. Halving the caption output cap from 1,0241{,}024 to 512512 tokens lowers EgoLifeQA accuracy from 40.6%40.6\% to 35.0%35.0\% (Table 9), while longer segments reduce QA under the fixed frame budget by allowing fewer independently selectable intervals (Figure 4). The revised manuscript foregrounds the construction objective in the Introduction and Section 3, and these trade-offs in Section 5.3 and Appendix D.2.


W3: This system has limited low-level visual retrieval capability. LoreMem relies heavily on frozen encoder priors and a fixed lexicon, limiting its ability to retrieve fine-grained visual details (e.g., clothing textures and materials). I do not consider it to be a major weakness since high-level semantic can slove plenty real life problems.

Thank you for highlighting fine-grained visual attributes. WordNet is a large English lexical database with broad coverage of nouns, verbs, adjectives, and adverbs [1]. Drawing on this resource, our lexicon contains 65,383 entries, including materials and fibers such as cotton, wool, silk, and leather; fabrics such as denim, velvet, and corduroy; textures such as rough, smooth, silky, and velvety; and textile structures such as knitted, woven, ribbed, and quilted. These entries provide explicit linguistic anchors for the clothing properties raised in the review.

LoreMem's motivation is to use a broad lexicon to make the semantic cues represented by frozen encoders accessible for indexing. Each frame is compared with visual lexicon anchors, and each segment retains its top-M concept identities and ranks. Queries select concepts from the same inventory, allowing material and texture terms to participate in retrieval through their stored ranks. This provides a shared concept interface for indexing visual attributes without generating descriptions.

The overall retrieval results support the utility of this lexicon-ordinal representation. Table 6 reports an average nDCG@20 of 18.8 for LoreMem, compared with 7.2 for the strongest visual-embedding baseline and 9.7 for local caption memory. Section 4.1 of the revised manuscript now specifies the lexicon's coverage of visual attributes and its indexing rationale, while distinguishing vocabulary coverage from reliable recognition by frozen encoders.

[1] Miller, G. A. (1995). WordNet: A Lexical Database for English. Communications of the ACM, 38(11), 39–41.

AuthorsOpenReview ↗

Kindly Follow-Up: Reviewer AR2p

Dear Reviewer AR2p,

Thank you again for your constructive review. As the discussion period ends in about two days, we would like to kindly follow up on our response above. We would greatly appreciate it if you could take a look, and we would be happy to clarify any remaining concerns.

Best regards, Authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We sincerely thank all reviewers for their constructive feedback and the AC for overseeing the discussion. We are encouraged that reviewers found a relevant and well-motivated problem (nohs, 9d8G, YHfz), a simple yet effective method (AR2p, nohs, YHfz), solid on-device evidence (AR2p, 9d8G, nohs), and consistent gains (nohs, 9d8G). As our individual responses are lengthy, we highlight below the key new results addressing the main concerns.


1. All seven requested streaming baselines (9d8G, AR2p). With shared video, dialogue, and question inputs, LoreMem (49.4%) exceeds Flash-VStream, StreamForest, Dispider, VST, JoyAI, TimeChat-Online, and Memento by 5.2–13.4 points (Table 8). With the same answerer and 300-frame budget, it also outperforms uniform sampling over the entire eligible history (52.0% vs. 41.3%; Table 7), confirming that the gain comes from query-guided evidence selection rather than restricted baseline access.

2. Discriminativeness and retrieval traces (9d8G). Removing the 500 most frequent concepts per source leaves nDCG@20 nearly unchanged (21.5 →\rightarrow 21.4; Table 16). For three visually near-identical poker episodes, each question retrieves its own episode at rank 1, 2, and 1 among 376 candidates (Figure 11). LoreMem's retrieval is also less concentrated than caption retrieval (top-1% share 12.52% vs. 17.08%) while achieving 4x higher nDCG@20. Figures 7-10 now trace the full query →\rightarrow concept →\rightarrow rank →\rightarrow segment pipeline, including a failure case.

3. End-to-end online evaluation (AR2p, YHfz). Retrieval now uses only memory whose construction has finished by query time, including the latency of answering earlier queries. LoreMem retains its 52.0% average QA, whereas Qwen3-VL-4B caption memory drops from 44.1% to 40.7%, widening the gap from 7.9 to 11.3 points.

4. Faster on-device caption builders (nohs). We timed Qwen3-VL-4B (512 tokens), Qwen3-VL-2B, and Qwen3.5-0.8B. The fastest builder (13.51 s per 30-s segment) falls below the arrival interval but reaches only 34.6% QA / 3.4 nDCG@20, versus LoreMem's 49.4% / 15.1 at 0.06 s (Table 9).

5. Value of the lexicon-ordinal index (nohs, YHfz). Against lexicon-free BGE/PE rank fusion, LoreMem raises mean nDCG@20 from 14.3 to 18.8, with significant gains on all four benchmarks under paired randomization tests (p < 0.05; Table 14). Dialogue-only LoreMem (λ=0\lambda=0) also improves over direct dialogue retrieval (11.9 →\rightarrow 16.8).

6. Lexicon source and coverage (9d8G, nohs, AR2p). The 65,383 entries are taken directly from the Open English Wordnet 2025 release, without dataset-specific selection. An inventory ablation (Table 10) shows the full lexicon performs best (49.4%), with nouns alone reaching 48.2%. For Chinese TeleEgo, translating inputs to English changes QA only marginally (54.7% →\rightarrow 55.2%), indicating that the cross-lingual mapping already works well.


Overall, the new evidence strengthens our core claim: LoreMem builds reusable memory over 600x faster than the default local captioner (0.06 s vs. 37.84 s per segment) while outperforming the evaluated on-device caption builders and streaming memory systems, and its advantage grows once construction delay is accounted for.

With about two days left in the discussion period, we would greatly appreciate it if reviewers could let us know whether our responses address their concerns or if any point needs further clarification. We remain available to provide additional results or analyses in the remaining time. If our responses resolve your concerns, we would be grateful if you would consider reflecting this in your evaluation.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.