acceptodds
Under review as a conference paper at ICLR 2027

Efficient Multimodal Memory Construction for Long-Form Video Retrieval

Abstract

Long-form video question answering requires retaining sparse evidence across extended temporal contexts. Retrieval-based systems make this setting tractable, but they can only retrieve evidence that was preserved when the video memory was written. Under a lightweight memory builder, compressing each clip into one query-agnostic caption is a difficult target: a generic narrative can omit fine-grained cues that later become relevant. We therefore propose **MVP-Mem**, a retrieval-oriented framework that represents each clip as short phrases extracted under complementary semantic views. The resulting phrases form compact, independently retrievable evidence anchors, accessed through a simple phrase-wise maximum scoring rule. Our primary comparison uses Qwen3-VL-4B for both caption and phrase memory construction, with the same downstream answer generator. Across four long-form video QA benchmarks, **MVP-Mem** improves over this caption baseline on every benchmark, raising average accuracy from 44.5% to 54.0% (+9.5 percentage points). On EgoLifeQA, it also nearly halves memory construction time relative to caption generation with the same Qwen3-VL-4B builder. These results support multi-view phrase memory as an effective construction target for lightweight VLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
efficient memory construction, long-form video QA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_17GbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-view, phrase-level caption generation approach that transforms text-to-video retrieval into text-to-phrase retrieval, thereby improving the question answering (QA) accuracy in downstream tasks.

Critical strengths

This paper provides a clear description of the methodology and a comprehensive experimental setup.

Critical weaknesses
  1. Structural and Contribution Issues: The structural organization of this paper is problematic. The core contribution should be framed around the synergistic design of both multi-view and phrase-level representations. However, the narrative focuses almost exclusively on the phrase-level aspect, lacking sufficient and rigorous discussion on the multi-view mechanism. Furthermore, the paper inappropriately presents ColBERT's MaxSim operation as one of its own core contributions, which is misleading.

  2. Questionable Experimental Data in Table 3: The experimental data in Table 3 raises concerns. First, it is unclear why there is a discrepancy in retrieval speed between standard single-vector retrieval and the single-vector retrieval of multi-phrases (which should theoretically incur identical online retrieval costs if vectors are precomputed). Second, the proposed retrieval method is fundamentally similar to ColBERT, yet there is a significant and unexplained discrepancy in the reported retrieval speeds. Essentially, the proposed retrieval method is merely ColBERT stripped of its query-side token-level decomposition, which does not constitute a novel methodological contribution, as noted in Point 1.

  3. Potential Benchmark Hacking and Insufficient Retrieval Evaluation: Relevant studies indicate that in QA tasks, retrieval precision and end-to-end QA accuracy do not necessarily have a strong correlation. This is particularly concerning here, as highly similar phrases from completely unrelated contexts might be retrieved simultaneously, potentially misleading the downstream generator. Since this paper primarily optimizes the retrieval stage, the multi-view setup might be susceptible to benchmark hacking, artificially inflating the QA scores (which ties back to the lack of rigorous multi-view discussion mentioned in Point 1). However, the experiments in Table 7, which are intended to demonstrate genuine improvements in retrieval precision, are somewhat insufficient to definitively rule out this possibility or provide a robust analysis of retrieval quality.

Questions for authors

The baselines selected in this paper appear to be predominantly from about a year ago. Have any more recent works been considered for comparison?

7 comments
AuthorsOpenReview ↗

Author's Response to Reviewer 17Gb (1/5)

We sincerely thank the reviewer 17Gb for their insightful and constructive comments. We have carefully considered the reviewer's concerns and provide our detailed responses below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

[W1] Structural and Contribution Issues: The structural organization of this paper is problematic. The core contribution should be framed around the synergistic design of both multi-view and phrase-level representations. However, the narrative focuses almost exclusively on the phrase-level aspect, lacking sufficient and rigorous discussion on the multi-view mechanism. Furthermore, the paper inappropriately presents ColBERT's MaxSim operation as one of its own core contributions, which is misleading.

Thank you for suggesting that we make the joint role of multi-view construction and phrase-level memory clearer. MVP-Mem's contribution is the joint design of view-conditioned memory writing and phrase-level evidence, rather than the maximum-similarity operator. Complementary semantic views decompose memory writing into focused extraction tasks, while compact phrases preserve their outputs as separately retrievable units. Treating the method as maximum scoring over generic text overlooks this construction design. We have reorganized Section 1 and Sections 2.2–2.4 to make this contribution explicit.

A memory builder must preserve potentially useful evidence without knowing future queries. Generic captioning asks the builder to identify salient observations, cover heterogeneous evidence, and integrate these observations into fluent narration within a single generation task. Our approach changes this task formulation rather than simply assigning the same captioning task to a smaller model. Multi-view construction narrows each extraction task to a specified evidence type, while the views collectively target complementary information, including actions, object states and locations, temporal cues, and object interactions. The purpose is therefore not merely to generate more content, but to make the extraction requirements explicit and reduce reliance on a single open-ended description.

Phrase-level output complements this decomposition. Once an extraction task is focused on a particular evidence type, the builder need not turn each observation into a complete sentence or combine heterogeneous observations into a coherent narrative. Instead, it produces compact phrases that retain local relational information beyond isolated object labels. Multi-view construction simplifies what each extraction task asks the model to identify; phrase-level output simplifies how the extracted evidence must be expressed. These are two coordinated choices for adapting memory writing to a lightweight builder, rather than independent additions to a generic captioning pipeline.

The resulting memory also supports selective access. Different phrases may describe different aspects of a clip, so aggregating them into one vector can obscure the evidence relevant to a particular query. We therefore mean-pool token representations within each phrase and retain separate phrase vectors for query-dependent scoring. Pooling within phrases preserves the chosen evidence unit, while keeping phrases separate allows retrieval to select among the collected observations. Maximum-similarity scoring serves this access requirement; it is not the source of our novelty claim. Section 2.4 now explicitly acknowledges ColBERT and ColBERTv2 and states that we do not claim novelty for the operator, retrieval backbone, or training objective; Section 4 provides the corresponding late-interaction context.

MemorySemantic viewsRetrievalGenerated tokens/clipIndexed vectors/clipQA accuracy (%)
CaptionNoSingle vector4101.040.6
Caption split into sentencesNoSentence-wise maximum41020.938.2
Single-view phrasesNoSingle vector471.048.0
Expanded single-view phrasesNoSingle vector2541.049.8
Expanded single-view phrasesNoPhrase-wise maximum25469.351.2
Multi-view phrasesYesSingle vector2131.052.0
Multi-view phrasesYesPhrase-wise maximum21349.054.4
AuthorsOpenReview ↗

Author's Response to Reviewer 17Gb (2/5)

Table 3 supports this design through comparisons that separate memory writing from the retrieval rule:

  • Phrase writing helps without multi-view construction or maximum scoring. With single-vector retrieval fixed, replacing caption memory with single-view phrase memory improves accuracy from 40.6% to 48.0%, while reducing generated tokens per clip from 410 to 47.
  • Structured multi-view extraction helps beyond increasing single-view output. Multi-view phrases outperform expanded single-view phrases under both single-vector retrieval (52.0% vs. 49.8%) and phrase-wise maximum scoring (54.4% vs. 51.2%), despite using fewer generated tokens per clip (213 vs. 254).
  • Maximum scoring alone does not explain the gains. Phrase-wise access improves the same multi-view memory from 52.0% to 54.4%, whereas sentence-wise maximum scoring over caption memory reduces accuracy from 40.6% to 38.2%.

These controls show that the gain cannot be explained by maximum scoring or increased output quantity alone. They support the joint memory-writing and retrieval design under the fixed lightweight builder. Sections 1 and 2.2–2.4 now foreground this contribution, Section 3.3 explains the controlled ablation in Table 3, and Appendix B.8 reports further view-count and induction-model sensitivity experiments.


[W2] Questionable Experimental Data in Table 3:
The experimental data in Table 3 raises concerns. First, it is unclear why there is a discrepancy in retrieval speed between standard single-vector retrieval and the single-vector retrieval of multi-phrases (which should theoretically incur identical online retrieval costs if vectors are precomputed). Second, the proposed retrieval method is fundamentally similar to ColBERT, yet there is a significant and unexplained discrepancy in the reported retrieval speeds. Essentially, the proposed retrieval method is merely ColBERT stripped of its query-side token-level decomposition, which does not constitute a novel methodological contribution, as noted in Point 1.

We apologize for the confusion caused by our original latency measurements. We identified and corrected the timing issue, and added both a component-wise breakdown and a comparison by search-pool size. The corrected aggregate costs are in Table 4; Appendix B.6 and Tables 10 and 11 report the protocol and the two analyses.

Single-vector retrieval. We mistakenly included the initial executions in the timing without a warm-up phase, which introduced variability into the original measurements (0.12 s and 0.08 s, each with a standard deviation of approximately 0.08 s). We corrected this by performing 100 warm-up iterations before measuring subsequent executions. Both single-vector configurations then yielded 0.07 ±\pm 0.01 s. We have corrected Table 4 accordingly and documented the warm-up procedure in Appendix B.6.

Explaining the latency difference from ColBERT. We measured two online stages separately: query embedding and per-clip scalar scoring over the eligible clip pool (Table 10). The reported latency in Tables 4, 10, 11, and 12 is the sum of these two measured stage times.

Measured stageColBERT (s)Phrase MaxSim (s)
Query embedding0.050.05
Per-clip scalar scoring0.360.21
Sum of measured stages0.410.26

Precomputed memory vectors do not eliminate online query–memory interaction. ColBERT aggregates token-level matches for multiple query vectors, whereas phrase-wise MaxSim compares one query vector with the clip's phrase vectors. The measured 0.15 s gap arises in clip scalar scoring (0.36 vs. 0.21 s); query embedding takes 0.05 s for both methods.

Table 4 reports standard deviations of 0.27 s for ColBERT and 0.38 s for phrase-wise MaxSim for the latency summed over the two measured stages. Queries have different retrieval-pool sizes, so these aggregate timings include workload variation; Appendix B.6 and Table 10 state this measurement scope.

Latency by searchable clips per question. Table 11 reports the directly measured mean latency covering query embedding and clip scalar scoring for both ColBERT and MVP-Mem after 100 warm-up iterations for the same EgoLifeQA question groups, defined by the number of searchable clips. Latency entries are displayed to two decimal places.

Searchable clips per questionColBERT latency (s)MVP-Mem latency (s)
Fewer than 2,0000.120.09
2,000–2,9990.370.24
3,000–3,9990.560.35
4,000–4,9990.690.42
5,000 or more0.860.52
AuthorsOpenReview ↗

Author's Response to Reviewer 17Gb (3/5)

MVP-Mem has lower measured latency than ColBERT in every search-pool group. Both methods' measured latency increases across groups with more searchable clips, while the absolute difference grows from 0.03 s in the smallest group to 0.34 s in the largest (0.52 s for MVP-Mem versus 0.86 s for ColBERT). Together with the measured aggregate comparison and component-wise breakdown in Tables 4 and 10, these results show a consistent latency advantage in the evaluated implementation. Each group contains different questions, so the trend describes measured performance across the observed workloads.

Scope of the contribution. The characterization of MVP-Mem as merely ColBERT with query-side decomposition removed misses the memory-construction component. Our lightweight 4B VLM directly writes view-conditioned phrases, and each generated phrase becomes a retrieval unit. The established MaxSim operator provides access to this memory. Sections 1 and 2.4 explicitly locate the contribution in memory writing and representation.

The revised Section 3.4 and Appendix B.6 report the corrected timings in Table 4, the component-wise breakdown in Table 10, and the search-pool comparison in Table 11. Sections 1 and 2.4 clarify that the contribution lies in lightweight memory construction rather than a new late-interaction scoring rule.


[W3] Potential Benchmark Hacking and Insufficient Retrieval Evaluation: Relevant studies indicate that in QA tasks, retrieval precision and end-to-end QA accuracy do not necessarily have a strong correlation. This is particularly concerning here, as highly similar phrases from completely unrelated contexts might be retrieved simultaneously, potentially misleading the downstream generator. Since this paper primarily optimizes the retrieval stage, the multi-view setup might be susceptible to benchmark hacking, artificially inflating the QA scores (which ties back to the lack of rigorous multi-view discussion mentioned in Point 1). However, the experiments in Table 7, which are intended to demonstrate genuine improvements in retrieval precision, are somewhat insufficient to definitively rule out this possibility or provide a robust analysis of retrieval quality.

We appreciate the reviewer's concern that improved downstream QA accuracy does not necessarily establish better retrieval quality, and that domain-specific view induction could introduce evaluation-dependent bias. We address these concerns by clarifying the view-induction protocol and providing retrieval-only and controlled distractor evaluations.

(1) Domain-conditioned view induction versus benchmark-specific optimization.

MVP-Mem uses high-level benchmark descriptions to define the types of evidence that the memory builder should preserve. These descriptions characterize the task domain, such as everyday activities, object locations, temporal relations, and open-domain video understanding, rather than the content of individual evaluation questions.

Crucially, the semantic views are induced once from these descriptions, fixed before clip memory construction, and applied uniformly to all clips within each benchmark. Individual test questions, answer choices, answer distributions, and ground-truth annotations are never provided during view induction. All generated views are retained without manual filtering or evaluation-based selection.

This protocol distinguishes domain-conditioned memory construction from optimization using evaluation-specific questions or answers. We acknowledge that the current method employs benchmark-level domain conditioning; transferring a single fixed schema across different domains is not evaluated and remains an important direction for future work. Section 2.3 and Appendix E.1 (Tables 19, 21, and 23) now explicitly document the induction procedure and the information provided to the cloud LLM.

(2) Retrieval quality independent of downstream QA.

To assess whether the improvements reflect better evidence retrieval rather than answer-generation effects, we report retrieval-only NDCG@20 in Table 8. With the same Qwen3-VL-4B memory builder, MVP-Mem improves NDCG@20 over caption memory from 3.7 to 15.9 on EgoLifeQA, 2.3 to 22.9 on Ego-R1, and 17.1 to 22.8 on SuperMemory-VQA.

These improvements are measured independently of the answer generator and directly demonstrate better ranking of ground-truth clips. They therefore provide evidence for the retrieval contribution beyond end-to-end QA accuracy.

Reviewer_X9W9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces MVP‑Mem, a retrieval‑focused multimodal memory framework for long‑form video QA. It addresses the bottleneck that lightweight VLMs struggle to produce high‑quality universal clip captions, which often miss fine‑grained retrieval‑critical evidence. MVP‑Mem first uses a cloud LLM to generate domain‑specific complementary semantic views in a one‑time offline step. A lightweight VLM extracts short retrieval‑oriented phrases under each view, with KV‑cache reuse to cut inference cost. During retrieval, a phrase‑wise maximum‑similarity scoring rule ranks clips by their best‑matching phrase embedding instead of averaging signals. Evaluated on four long‑video QA benchmarks, using Qwen3‑VL‑4B for memory construction, MVP‑Mem improves average QA accuracy from 44.5% to 54.0% versus single‑caption baselines, while lowering offline memory‑construction tokens and runtime. Controlled ablations validate the benefits of phrase‑level representation, multi‑view decomposition, and the max‑sim scoring rule. The method trades increased retrieval‑side index overhead for better end‑task performance.

Critical strengths

The paper draws attention to an under‑studied bottleneck for lightweight‑VLM‑powered long‑video RAG: generating a single all‑purpose caption per clip is a difficult target for small models, which risks losing fine‑grained retrieval‑relevant evidence. Rather than proposing a complex new retriever, it revises the output target for offline memory building. This is a practically meaningful perspective for egocentric / wearable long‑video use‑cases, and sets a clear motivation for the proposed MVP‑Mem design.

Critical weaknesses
  1. The work adopts a simple phrase‑wise Max‑Sim aggregation rule for clip scoring. However, this aggregation cannot handle queries that require combining multiple weak signals or performing compositional reasoning across different phrases. Besides, the sensitivity analysis for key hyper‑parameters is limited, including the retrieval top‑K value, the number of semantic views, as well as the trade‑off between index growth and accuracy under large‑scale settings with numerous videos and users.

  2. The paper attributes performance gains to complementary semantic decomposition enabled by multi‑view design. Nevertheless, several critical ablations are missing to validate this claim. First, the performance fluctuation caused by different sets of semantic views induced by distinct cloud‑based LLMs is not evaluated. Second, controlled experiments such as randomly removing partial views or injecting redundant noisy views are absent. Therefore, it is hard to conclusively verify that the improvement stems from complementary semantic decomposition, rather than merely the increased number of phrases and embedding vectors.

  3. Several recent relevant works on retrieval‑aware memory construction and structured / event‑centric video memories are only superficially touched upon or entirely omitted, including EM2Mem [1], R4DSG [2], MERIT [3], and DigimonGPT [4]. More systematic discussion against these approaches would help better clarify the niche of MVP‑Mem.

[1] EM^2Mem: Event-Centric Multimodal Memory for Large Language Models [2] R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video [3] Keep It Simple: Multi-key Episodic Memory Retrieval for Ultra-Long Video Understanding [4] DigimonGPT: An Evolvable Agent with Hierarchical Human-like Memory for Video Question Answering

Questions for authors

Refer to the weakness section.

9 comments
AuthorsOpenReview ↗

Author's Response to Reviewer X9W9 (1/4)

We sincerely thank the reviewer X9W9 for their constructive feedback and recognition of the practical significance and motivation behind our MVP-Mem design. We address the reviewer's concerns in detail below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

[W1] Limitations of Phrase-Wise Aggregation and Underexplored Accuracy–Index Size Trade-offs The work adopts a simple phrase‑wise Max‑Sim aggregation rule for clip scoring. However, this aggregation cannot handle queries that require combining multiple weak signals or performing compositional reasoning across different phrases. Besides, the sensitivity analysis for key hyper‑parameters is limited, including the retrieval top‑K value, the number of semantic views, as well as the trade‑off between index growth and accuracy under large‑scale settings with numerous videos and users.

MaxSim is the clip-ranking rule, not the answer-reasoning mechanism. It selects the strongest query–phrase match for each clip; the answer generator then receives the top-K clips and can combine their evidence. The absence of explicit compositional aggregation in the scoring rule therefore does not establish that the full pipeline cannot answer compositional questions. Designing an explicit multi-hop retrieval controller is outside this paper's scope of efficient, query-independent memory construction. Query decomposition offers a compatible extension: Question Decomposition for Retrieval-Augmented Generation (Ammann et al., 2025) retrieves evidence for decomposed sub-questions, while IRCoT (Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions, Trivedi et al., 2023) interleaves retrieval with reasoning in text-based multi-step QA.

A controller could decompose a complex video question into subqueries, retrieve clips for each using the same phrase memories, and merge or iteratively update the evidence before answering. This extends query-time evidence collection while retaining our memory-construction design. Appendix D explicitly describes the scoring limitation and this integration as future work; our current results concern the evaluated single-round pipeline.

Top-K sensitivity is already evaluated in Figure 9, which compares K=10, 20, and 50 on EgoLifeQA. MVP-Mem outperforms the compared caption and visual memories across those settings. These are end-to-end budget comparisons: increasing K also increases the evidence supplied to the answer model.

Figure 3(b) already evaluates view count through random removal from the original 15-view schema. The 5- and 10-view settings retain randomly selected subsets of the original views, removing 10 and 5 views, respectively. With 5, 10, and all 15 semantic views, MVP-Mem achieves 51.6%, 52.4%, and 54.4% on EgoLifeQA; the figure also reports their construction-time trade-off. Figure 10 in Appendix B.4 relates these same settings to generated-token cost. For this rebuttal, we conducted additional experiments with 20 and 25 views. Appendix B.8 and Table 13 now consolidate the existing ablation and these new results.

Number of semantic viewsEgoLifeQA accuracy (%)
551.6
1052.4
1554.4
2056.1
2556.5

The 20- and 25-view settings augment the original schema with additional induced views. The gains are +1.7 pp from 15 to 20 views and +0.4 pp from 20 to 25, showing diminishing incremental improvement in the tested range. The default 15 views came from the original unconstrained induction, not accuracy-based tuning of the view count. Appendix B.8 and Table 13 report the sensitivity results; Section 2.3 and Table 1 describe the original main configuration.

Construction and retrieval costs. On EgoLifeQA, MVP-Mem uses 213 generated tokens per clip versus 410 for captions from the same 4B builder and takes 7.7 s versus approximately 16 s to construct. KV-cache reuse reduces 15-view construction from 21.5 s to 7.7 s (Figure 5). Retaining separate phrase vectors adds storage and retrieval work; Appendix B.7 and Table 12 report the resulting QA, storage, and latency trade-off across search-pool sizes, reproduced below. The corrected aggregate timings and their variability are reported in Table 4 and explained in Appendix B.6 and Table 10. Appendix B.6 and Table 11 additionally report directly measured ColBERT and MVP-Mem latencies for the same search-pool groups.

AuthorsOpenReview ↗

Author's Response to X9W9 (2/4)

Joint analysis of search-pool size, QA, storage, and latency. We appreciate the suggestion to evaluate accuracy alongside storage and latency. We grouped EgoLifeQA questions by the number of clips eligible for retrieval and compared MVP-Mem with GPT-5.2 caption memory, the stronger-caption reference in Table 7. Appendix B.7 and Table 12 now jointly report memory storage in MB, QA accuracy, and measured latency covering query embedding and clip scalar scoring in seconds for each group:

Searchable clips per questionMVP-Mem storage (MB)MVP-Mem QA (%)MVP-Mem latency (s)GPT-5.2 caption storage (MB)GPT-5.2 caption QA (%)GPT-5.2 caption latency (s)
Fewer than 2,000176.6659.340.093.6258.240.07
2,000–2,999471.4151.820.249.6647.270.07
3,000–3,999691.0151.190.3514.1634.520.07
4,000–4,999844.2457.140.4217.3037.500.07
5,000 or more1,042.8647.060.5221.3733.820.07

MVP-Mem outperforms GPT-5.2 caption memory in every search-pool group, with a larger advantage in groups containing at least 3,000 searchable clips. The gain is +1.10 and +4.55 pp in the two smaller-pool groups, compared with +13.24 to +19.64 pp in the larger-pool groups. Overall QA accuracy is 54.40% versus 46.20% (+8.20 pp). Between the smallest and largest groups, MVP-Mem's accuracy is lower by 12.28 pp, while GPT-5.2 caption accuracy is lower by 24.42 pp. This pattern is consistent with MVP-Mem retaining more of its QA performance in larger search pools, although the gap does not increase monotonically across every group.

The table also quantifies the storage and latency cost of this QA advantage. Storage denotes uncompressed FP32 vector tensors in our PyTorch implementation, with matched embedding dimensionality and dtype for both representations. The vector payload scales linearly with the total number of stored phrases, and MVP-Mem uses approximately 48.8 times the vector storage of caption memory. In the largest group, MVP-Mem achieves a +13.24 pp QA gain with 1,042.86 MB of vector storage and 0.52 s retrieval latency. These results make the accuracy–storage–latency trade-off explicit over the evaluated search-pool sizes. Storage could be reduced through product quantization supported by Faiss(Douze et al., IEEE Transactions on Big Data, 2025) or residual compression as used in ColBERTv2(Santhanam et al., NAACL, 2022). offer promising directions for substantially reducing this uncompressed FP32 storage footprint.

Appendix B.7 and Table 12 directly address the requested accuracy–index-size–latency trade-off over the evaluated search pools. The groups contain different questions, so they characterize the observed collection rather than a controlled expansion for fixed queries. Pooled multi-user retrieval and further corpus expansion are outside the evaluated dataset setting; Appendix D identifies these extensions as future work.

AuthorsOpenReview ↗

Author's Response to Reviewer X9W9 (3/4)

[W2] Insufficient Ablation Studies to Validate Complementary Semantic Decomposition The paper attributes performance gains to complementary semantic decomposition enabled by multi‑view design. Nevertheless, several critical ablations are missing to validate this claim. First, the performance fluctuation caused by different sets of semantic views induced by distinct cloud‑based LLMs is not evaluated. Second, controlled experiments such as randomly removing partial views or injecting redundant noisy views are absent. Therefore, it is hard to conclusively verify that the improvement stems from complementary semantic decomposition, rather than merely the increased number of phrases and embedding vectors.

Random view removal is already evaluated in Figure 3(b), and the output-quantity explanation is tested by the expanded single-view control in Table 3. The randomly selected 5- and 10-view subsets remove 10 and 5 views from the original schema, respectively. Appendix B.8 and Table 13 consolidate these results with the additional 20/25-view experiments. For this rebuttal, we also regenerated 15-view schemas with two additional cloud LLMs, keeping the meta-prompt and downstream builder, retriever, answer generator, and other settings fixed. Appendix B.8 and Table 14 report:

View-induction modelViewsEgoLifeQA accuracy (%)
GPT-5.21554.4
Claude 4.6 Sonnet1552.6
Gemini 3.1 Pro1552.8

All generated views were retained without manual filtering. The three schemas yield accuracies within 1.8 pp, showing that the result is not restricted to the exact original schema or inducer. Table 14 reports one induced schema per model, as stated in Appendix B.8 and the caption.

The output-quantity control in Table 3 addresses whether the gain comes merely from more phrases or vectors. With phrase-wise maximum fixed, multi-view memory achieves 54.4% versus 51.2% for expanded single-view memory while using fewer tokens (213 versus 254) and indexed vectors (49.0 versus 69.3). With single-vector retrieval fixed, the corresponding accuracies are 52.0% and 49.8%. We conducted this expanded single-view control as an additional experiment for this rebuttal and incorporated its results into Section 3.3 and Table 3 of the revised manuscript. Thus, increased output quantity alone does not explain the observed ordering.

The combined evidence addresses view removal, inducer dependence, and output quantity; the gain is not explained merely by more phrases or embedding vectors. Relative to all 15 views, random 10- and 5-view subsets reduce accuracy by 2.0 and 2.8 pp. Table 3 separately shows that multi-view memory outperforms a larger single-view memory under both retrieval rules. Deliberately injecting redundant or noisy instructions is a different perturbation from view removal and lies outside the evaluated fixed-schema protocol. Appendix B.8 distinguishes these conditions, and Appendix D lists noisy-view evaluation as future work.

Reviewer_z5WdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies efficient memory construction for long-form video retrieval. It discusses the limitations of single-vector embeddings and caption-based memories for retrieval. The proposed method, MVP-Mem, represents each video clip as a multi-view phrase memory, in which phrases extracted under complementary semantic views serve as independently retrievable evidence anchors. These anchors are accessed through a phrase-wise maximum scoring rule. The method is evaluated on four long-form video QA benchmarks and improves over the baselines.

Critical strengths
  • The paper clearly discusses the limitations of existing single-embedding and caption-based retrieval approaches.

  • The proposed phrase-level memory and retrieval scheme is simple and practical.

Critical weaknesses
  • The paper provides only one qualitative example of constructed memories (Fig. 6, Appendix) and no examples of retrieved clips. More examples of phrase memories, retrieval successes and failures, and comparisons with the caption baseline would help readers understand how the method works and where it fails.

  • Key components, namely the complementary semantic views and the view-specific prompts, are not clearly described in the main paper. They are given only in the appendix. The main text should explain what the views are, how many are used for each benchmark, and how they differ across benchmarks.

  • It is unclear how the method ensures that the constructed phrases are factually correct for each clip. A lightweight VLM can hallucinate, and the paper itself notes this as a limitation of captions. Phrase memories, however, are not evaluated for factual accuracy.

  • The technical contribution is limited. Memory construction amounts to prompting an existing lightweight VLM (Qwen3-VL-4B) with a set of view-specific prompts. The retrieval side relies on the simple maximum-similarity scoring rule. The method thus mainly combines existing models with prompt engineering, without new model design, training objectives, or theoretical insight. This limits its contribution to the research field.

Questions for authors
  • How sensitive is the method to the quality of clip partitioning?
  • How do the results change with different clip lengths or with event-based segmentation?
8 comments
AuthorsOpenReview ↗

Author's Response to Reviewer z5Wd (1/4)

We sincerely thank the reviewer for their constructive feedback and recognition of our clear motivation and the simplicity and practicality of the proposed phrase-level memory and retrieval scheme. We address the reviewer's concerns in detail below.

Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.

[W1] Insufficient Qualitative Examples of Memory Construction and Retrieval Performance The paper provides only one qualitative example of constructed memories (Fig. 6, Appendix) and no examples of retrieved clips. More examples of phrase memories, retrieval successes and failures, and comparisons with the caption baseline would help readers understand how the method works and where it fails.

Thank you for suggesting additional examples of retrieval successes and failures. Appendix B.1 now includes retrieval success and failure examples in Figures 7 and 8, in addition to the construction examples in Figure 6. Each new figure shows the query, ground-truth (GT) clip and its ASR transcript, retrieved clips, and the GT memory under MVP-Mem and GPT-5.2 captioning. Figure 7 displays ranks 1–3 for both methods; Figure 8 displays MVP-Mem ranks 1–3 and caption ranks 1, 2, and 7 (GT).

Success case (Fig. 7). For the query "How much is an ego coin worth in RMB?", MVP-Mem retrieves the GT clip at Rank 1. Its phrase memory stores the key fact as short phrases that match the query, such as "3 RMB", "set price at 3 RMB", and "determining ego coin price". The GPT-5.2 caption also contains the answer, but it appears inside a 272-word narrative with a lot of stylized content unrelated to the query. This weakens the match between the caption and the query, so the baseline does not rank the GT clip in its top 3. This example illustrates our main motivation: phrase-level memory keeps fine-grained facts easy to retrieve, while long free-form captions tend to bury them.

Failure case (Fig. 8). For the query "When was the doll in Katrina's hand grabbed from the claw machine?", MVP-Mem's top three results are neighboring clips and do not include the GT clip. All three are matched by the same generic phrase, "claw machine", which appears throughout the arcade sequence and cannot tell nearby moments apart. The GT phrase memory contains "catching prize", but lacks sufficiently specific context linking the event to the queried doll, person, and moment. It also contains a misinterpreted phrase ("need to catch coins"). The caption baseline keeps the dialogue context ("decoration for the house") and ranks the GT clip at Rank 7. This case illustrates how generic matches and insufficient contextual specificity can hinder precise temporal localization even when a related event phrase is present.

These examples show both the strength of compact phrase memory for retrieving specific facts and its weakness with evidence that depends on context. The success and failure analyses are now included in Appendix B.1 with Figures 7 and 8, and Appendix D discusses the loss of conversational context as a limitation.


[W2] Unclear Description of Semantic Views and View-Specific Prompts Key components, namely the complementary semantic views and the view-specific prompts, are not clearly described in the main paper. They are given only in the appendix. The main text should explain what the views are, how many are used for each benchmark, and how they differ across benchmarks.

Section 2.3 now defines the view mechanism in the main text, and Table 1 gives the benchmark-specific counts and evidence categories. A semantic view is a fixed instruction targeting a category of clip evidence. A cloud LLM induces the schema once from a high-level domain description, and the lightweight VLM applies its fixed instructions uniformly to every clip. Views are not regenerated for individual queries.

Section 2.3 and Table 1 now state the benchmark-specific view counts and evidence categories:

BenchmarkNumber of viewsMain evidence categories
EgoLifeQA and Ego-R115 shared viewsObject traces/state, actions and interactions, dialogue intentions/plans, routines, task progress, and temporal/causal cues.
Video-MME (Long)12Open-domain actions/motion, spatial layout, OCR, counting, synopsis, and unusual events.
SuperMemory-VQA18Personal belongings, conversations, intentions, locations, temporal anchors, and distinctive evidence-linking entities.
AuthorsOpenReview ↗

Author's Response to Reviewer z5Wd (2/4)

For example, the EgoLife Object Interaction Trace view uses video to extract handling/transfer phrases, with prompt examples such as "picked up screwdriver" and "moved backpack to sofa." The Dialogue Intention and Plans view uses speech text to extract intentions or future tasks, with examples such as "need to buy milk." These are output-format examples from the fixed prompts, rather than new observed retrieval cases. Some views use video, some text, and others both, as specified in Appendix E.1, Tables 20, 22, and 24.

The view-wise lists are combined into a clip phrase pool, jointly encoded with the shared text encoder, and mean-pooled within each phrase span to obtain separate retrieval vectors (Section 2.4). View count is fixed for a schema, while phrase count varies by clip. The full EgoLifeQA configuration averages 49.0 indexed phrase vectors and 213 generated tokens per clip (Table 3).

We have added the view definition, counts, domain conditioning, modality assignments, and representative instructions to Section 2.3 and Table 1. The complete benchmark-specific schemas and templates remain available in Appendices E.1–E.2, Tables 19–25. The additional 20/25-view sensitivity results are now reported in Appendix B.8 and Table 13; they do not change the benchmark-specific main configurations above.


[W3] Lack of Factual Accuracy Evaluation for Constructed Phrase Memories It is unclear how the method ensures that the constructed phrases are factually correct for each clip. A lightweight VLM can hallucinate, and the paper itself notes this as a limitation of captions. Phrase memories, however, are not evaluated for factual accuracy.

Factual accuracy is an important point, and we appreciate the request to evaluate it directly. We added a video-grounded factuality evaluation on all 6,266 EgoLifeQA clips, using GPT-6.1-luna as the judge, for MVP-Mem and caption memories generated by the same Qwen3-VL-4B builder. For each clip, the judge receives video frames sampled at 0.5 fps, the corresponding dialogue, and the candidate memory text, and checks each generated unit against this evidence. Appendix C.2 specifies the protocol, Table 17 gives the complete judge prompt, and Table 18 reports the results.

The same judging prompt is used for both representations. It instructs the judge to assess each sentence or phrase independently and decompose it into atomic factual claims. A unit receives 1 only if every claim is explicitly shown in the video, stated in the dialogue, or directly entailed by the supplied evidence. If any claim conflicts with the evidence, cannot be verified, or remains uncertain, the unit receives 0. External knowledge and guesses are prohibited. The prompt also specifies checks for speaker attribution, distinctions between what is said and what is shown or happens, temporal order, and claims about emotions, intentions, or causes. The judge returns one binary score per input unit in JSON format.

For the Qwen3-VL-4B caption baseline, the evaluation units are individual sentences; for MVP-Mem, they are the generated phrases. For each clip, we calculated the number of units judged factually correct divided by the total number of generated units. We then took the arithmetic mean of these per-clip ratios. Let N=6266N=6266 denote the number of evaluated clips:

\frac{\text{Number of factually correct units in clip }c} {\text{Number of generated units in clip }c}.$$ Thus, the caption ratio uses factually correct sentences divided by all generated sentences in a clip, while the MVP-Mem ratio uses factually correct phrases divided by all generated phrases in that clip. **The reported score is a macro average over clips:** each clip receives equal weight, irrespective of its number of generated units. | Memory representation | Video-grounded factuality (%) | | --------------------------------------- | ----------------------------: | | Qwen3-VL-4B caption (sentence evaluation) | 93.57 | | MVP-Mem (phrase evaluation) | 97.74 | **MVP-Mem achieves 97.74% video-grounded factuality versus 93.57% for captions from the same builder, a gain of 4.17 pp (Table 18).** Appendix C.2 documents the evaluator, clip count, input modalities, sampling rate, and scoring rules; Table 17 supplies the actual prompt, and Equation (5) defines the macro average. This directly addresses factual correctness of the constructed memory and is separate from the GPT-5.2 coverage/discriminativeness diagnostic in Appendix C.1, Figure 4, and Table 16.
AuthorsOpenReview ↗

Author's Response to Reviewer z5Wd (3/4)

[W4] Limited Technical Novelty Beyond Existing Models and Prompt Engineering The technical contribution is limited. Memory construction amounts to prompting an existing lightweight VLM (Qwen3-VL-4B) with a set of view-specific prompts. The retrieval side relies on the simple maximum-similarity scoring rule. The method thus mainly combines existing models with prompt engineering, without new model design, training objectives, or theoretical insight. This limits its contribution to the research field.

We appreciate the reviewer's concern regarding technical novelty. Our contribution is not a new VLM architecture or similarity operator, but a retrieval-oriented memory-writing formulation that jointly designs what evidence a lightweight VLM extracts and how that evidence remains accessible for retrieval. Specifically, complementary semantic views decompose query-agnostic memory construction into focused extraction tasks, while phrase-level representations preserve the resulting observations as independently retrievable evidence units. This coordinated design addresses the difficulty of compressing heterogeneous clip information into a single caption under limited builder capacity.

Importantly, Table 3 provides controlled evidence that the improvement cannot be explained by increased output quantity or maximum-similarity scoring alone. With the memory builder and answer generator fixed, we observe three distinct effects:

  • Memory-writing formulation: Replacing captions with single-view phrases improves accuracy from 40.6% to 48.0%, even with the same single-vector retrieval strategy.
  • Complementary semantic decomposition: Multi-view phrases outperform expanded single-view phrases under both single-vector retrieval (52.0% vs. 49.8%) and phrase-wise maximum scoring (54.4% vs. 51.2%), despite using fewer generated tokens (213 vs. 254) and indexed vectors (49.0 vs. 69.3).
  • Phrase-level accessibility: With multi-view memory fixed, retaining separate phrase vectors further improves accuracy from 52.0% to 54.4%, demonstrating the benefit of selective access to the extracted evidence.

These findings establish the central design insight of MVP-Mem: under a lightweight memory builder, explicitly structuring evidence extraction across complementary semantic views is more effective than simply expanding a single-view description, and preserving phrase-level access provides an additional benefit. Thus, our contribution lies in the joint design of memory construction and representation, rather than in the individual use of prompts or MaxSim. We explicitly acknowledge the established nature of maximum-similarity retrieval and related multi-key approaches, including ColBERT, ColBERTv2, and MERIT, in Sections 2.4 and 4.

The practical significance of this design is further supported by results across four long-form video QA benchmarks. Using the same Qwen3-VL-4B memory builder and Qwen3-VL-8B answer generator, MVP-Mem improves average QA accuracy from 44.5% to 54.0% (+9.5 pp) over caption memory (See Table 2). On EgoLifeQA, it also reduces generated tokens per clip from 410 to 213 and construction time from approximately 16 s to 7.7 s, while incurring the separately reported retrieval-storage overhead. Furthermore, MVP-Mem achieves 54.0% average accuracy compared with 51.8% for GPT-5.2 caption memory (See Table 7), demonstrating the competitiveness of this lightweight construction approach against a substantially stronger caption writer.

We have revised Sections 1 and 5 to foreground this memory-writing contribution, Section 3.3 and Table 3 to isolate its constituent effects, and Section 3.4 to clarify the associated construction–retrieval trade-off.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We sincerely thank all reviewers for their constructive feedback and the AC for overseeing the discussion. We are especially grateful to Reviewer 17Gb for engaging with our rebuttal, acknowledging that their concerns have been addressed, and raising their score (2 →\rightarrow 3). Below, we summarize how the revision addresses the shared concerns. All changes are highlighted in blue in the revised manuscript.

1. Contribution clarity (17Gb, z5Wd)

Our contribution is a memory-writing formulation for lightweight builders. It jointly designs what evidence is extracted (complementary semantic views) and how it remains accessible (phrase-level units). We do not claim the MaxSim operator, which is now explicitly credited to ColBERT/ColBERTv2 (Sections 1, 2.4, 4). The controlled ablation in Table 3 separates these effects:

  • Phrase writing alone: 40.6% →\rightarrow 48.0%.
  • Multi-view vs. expanded single-view phrases: 54.4% vs. 51.2%, with fewer tokens (213 vs. 254).
  • MaxSim over caption sentences hurts accuracy: 40.6% →\rightarrow 38.2%.

Thus, the gains do not come from output quantity or MaxSim alone.

2. Robustness of the multi-view design (X9W9, 17Gb)

  • View count (Table 13): 51.6 / 52.4 / 54.4 / 56.1 / 56.5% for 5 / 10 / 15 / 20 / 25 views.
  • View-induction LLMs (Table 14): GPT-5.2, Claude 4.6 Sonnet, and Gemini 3.1 Pro fall within 1.8 pp.
  • No test data in induction: views are induced once from domain-level descriptions, without test questions or answers.

3. Efficiency and scalability (17Gb, X9W9)

  • Corrected latency with warm-up (Table 4) and a component-wise breakdown (Table 10).
  • Accuracy–storage–latency analysis by search-pool size (Tables 11–12): MVP-Mem beats GPT-5.2 caption memory in every group, by +13.2 to +19.6 pp at 3,000+ clips. We openly report the added vector storage and point to PQ and residual compression as mitigations.

4. Factuality, qualitative analysis, and partitioning (z5Wd)

  • Video-grounded factuality on all 6,266 EgoLifeQA clips (Table 18): 97.74% for MVP-Mem vs. 93.57% for same-builder captions.
  • Qualitative examples (Figs. 7–8): one success case and one failure case.
  • Partitioning (Table 15): the advantage holds for 30 s, 60 s, and 600 s clips and for event-based segmentation.

5. Related work (X9W9)

Section 4 now discusses EM2Mem, R4DSG, MERIT, and DigimonGPT, and clarifies that MVP-Mem writes view-conditioned phrases directly rather than extracting keys from captions.


To the remaining reviewers:

Reviewer X9W9: We would greatly appreciate it if you could check whether the following address your concerns:

  • the view-count and induction-model analyses (Tables 13–14);
  • the output-quantity control (Table 3);
  • the accuracy–storage–latency analysis (Table 12).

Reviewer z5Wd: We would particularly appreciate your reassessment of our technical contribution in light of the controlled ablation (Table 3), as well as your feedback on the following additional evaluations:

  • new factuality evaluation (Table 18);
  • the success and failure examples (Figs. 7–8);
  • the partitioning study (Table 15).

If any concerns remain, we would be glad to address them during the remaining discussion period.

Thank you again for your time and thoughtful reviews.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.