- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multi-view, phrase-level caption generation approach that transforms text-to-video retrieval into text-to-phrase retrieval, thereby improving the question answering (QA) accuracy in downstream tasks.
Critical strengths
This paper provides a clear description of the methodology and a comprehensive experimental setup.
Critical weaknesses
-
Structural and Contribution Issues: The structural organization of this paper is problematic. The core contribution should be framed around the synergistic design of both multi-view and phrase-level representations. However, the narrative focuses almost exclusively on the phrase-level aspect, lacking sufficient and rigorous discussion on the multi-view mechanism. Furthermore, the paper inappropriately presents ColBERT's MaxSim operation as one of its own core contributions, which is misleading.
-
Questionable Experimental Data in Table 3: The experimental data in Table 3 raises concerns. First, it is unclear why there is a discrepancy in retrieval speed between standard single-vector retrieval and the single-vector retrieval of multi-phrases (which should theoretically incur identical online retrieval costs if vectors are precomputed). Second, the proposed retrieval method is fundamentally similar to ColBERT, yet there is a significant and unexplained discrepancy in the reported retrieval speeds. Essentially, the proposed retrieval method is merely ColBERT stripped of its query-side token-level decomposition, which does not constitute a novel methodological contribution, as noted in Point 1.
-
Potential Benchmark Hacking and Insufficient Retrieval Evaluation: Relevant studies indicate that in QA tasks, retrieval precision and end-to-end QA accuracy do not necessarily have a strong correlation. This is particularly concerning here, as highly similar phrases from completely unrelated contexts might be retrieved simultaneously, potentially misleading the downstream generator. Since this paper primarily optimizes the retrieval stage, the multi-view setup might be susceptible to benchmark hacking, artificially inflating the QA scores (which ties back to the lack of rigorous multi-view discussion mentioned in Point 1). However, the experiments in Table 7, which are intended to demonstrate genuine improvements in retrieval precision, are somewhat insufficient to definitively rule out this possibility or provide a robust analysis of retrieval quality.
Questions for authors
The baselines selected in this paper appear to be predominantly from about a year ago. Have any more recent works been considered for comparison?
7 comments
Author's Response to Reviewer 17Gb (2/5)
Table 3 supports this design through comparisons that separate memory writing from the retrieval rule:
- Phrase writing helps without multi-view construction or maximum scoring. With single-vector retrieval fixed, replacing caption memory with single-view phrase memory improves accuracy from 40.6% to 48.0%, while reducing generated tokens per clip from 410 to 47.
- Structured multi-view extraction helps beyond increasing single-view output. Multi-view phrases outperform expanded single-view phrases under both single-vector retrieval (52.0% vs. 49.8%) and phrase-wise maximum scoring (54.4% vs. 51.2%), despite using fewer generated tokens per clip (213 vs. 254).
- Maximum scoring alone does not explain the gains. Phrase-wise access improves the same multi-view memory from 52.0% to 54.4%, whereas sentence-wise maximum scoring over caption memory reduces accuracy from 40.6% to 38.2%.
These controls show that the gain cannot be explained by maximum scoring or increased output quantity alone. They support the joint memory-writing and retrieval design under the fixed lightweight builder. Sections 1 and 2.2–2.4 now foreground this contribution, Section 3.3 explains the controlled ablation in Table 3, and Appendix B.8 reports further view-count and induction-model sensitivity experiments.
[W2] Questionable Experimental Data in Table 3:
The experimental data in Table 3 raises concerns. First, it is unclear why there is a discrepancy in retrieval speed between standard single-vector retrieval and the single-vector retrieval of multi-phrases (which should theoretically incur identical online retrieval costs if vectors are precomputed). Second, the proposed retrieval method is fundamentally similar to ColBERT, yet there is a significant and unexplained discrepancy in the reported retrieval speeds. Essentially, the proposed retrieval method is merely ColBERT stripped of its query-side token-level decomposition, which does not constitute a novel methodological contribution, as noted in Point 1.
We apologize for the confusion caused by our original latency measurements. We identified and corrected the timing issue, and added both a component-wise breakdown and a comparison by search-pool size. The corrected aggregate costs are in Table 4; Appendix B.6 and Tables 10 and 11 report the protocol and the two analyses.
Single-vector retrieval. We mistakenly included the initial executions in the timing without a warm-up phase, which introduced variability into the original measurements (0.12 s and 0.08 s, each with a standard deviation of approximately 0.08 s). We corrected this by performing 100 warm-up iterations before measuring subsequent executions. Both single-vector configurations then yielded 0.07 0.01 s. We have corrected Table 4 accordingly and documented the warm-up procedure in Appendix B.6.
Explaining the latency difference from ColBERT. We measured two online stages separately: query embedding and per-clip scalar scoring over the eligible clip pool (Table 10). The reported latency in Tables 4, 10, 11, and 12 is the sum of these two measured stage times.
| Measured stage | ColBERT (s) | Phrase MaxSim (s) |
|---|---|---|
| Query embedding | 0.05 | 0.05 |
| Per-clip scalar scoring | 0.36 | 0.21 |
| Sum of measured stages | 0.41 | 0.26 |
Precomputed memory vectors do not eliminate online query–memory interaction. ColBERT aggregates token-level matches for multiple query vectors, whereas phrase-wise MaxSim compares one query vector with the clip's phrase vectors. The measured 0.15 s gap arises in clip scalar scoring (0.36 vs. 0.21 s); query embedding takes 0.05 s for both methods.
Table 4 reports standard deviations of 0.27 s for ColBERT and 0.38 s for phrase-wise MaxSim for the latency summed over the two measured stages. Queries have different retrieval-pool sizes, so these aggregate timings include workload variation; Appendix B.6 and Table 10 state this measurement scope.
Latency by searchable clips per question. Table 11 reports the directly measured mean latency covering query embedding and clip scalar scoring for both ColBERT and MVP-Mem after 100 warm-up iterations for the same EgoLifeQA question groups, defined by the number of searchable clips. Latency entries are displayed to two decimal places.
| Searchable clips per question | ColBERT latency (s) | MVP-Mem latency (s) |
|---|---|---|
| Fewer than 2,000 | 0.12 | 0.09 |
| 2,000–2,999 | 0.37 | 0.24 |
| 3,000–3,999 | 0.56 | 0.35 |
| 4,000–4,999 | 0.69 | 0.42 |
| 5,000 or more | 0.86 | 0.52 |
Author's Response to Reviewer 17Gb (3/5)
MVP-Mem has lower measured latency than ColBERT in every search-pool group. Both methods' measured latency increases across groups with more searchable clips, while the absolute difference grows from 0.03 s in the smallest group to 0.34 s in the largest (0.52 s for MVP-Mem versus 0.86 s for ColBERT). Together with the measured aggregate comparison and component-wise breakdown in Tables 4 and 10, these results show a consistent latency advantage in the evaluated implementation. Each group contains different questions, so the trend describes measured performance across the observed workloads.
Scope of the contribution. The characterization of MVP-Mem as merely ColBERT with query-side decomposition removed misses the memory-construction component. Our lightweight 4B VLM directly writes view-conditioned phrases, and each generated phrase becomes a retrieval unit. The established MaxSim operator provides access to this memory. Sections 1 and 2.4 explicitly locate the contribution in memory writing and representation.
The revised Section 3.4 and Appendix B.6 report the corrected timings in Table 4, the component-wise breakdown in Table 10, and the search-pool comparison in Table 11. Sections 1 and 2.4 clarify that the contribution lies in lightweight memory construction rather than a new late-interaction scoring rule.
[W3] Potential Benchmark Hacking and Insufficient Retrieval Evaluation: Relevant studies indicate that in QA tasks, retrieval precision and end-to-end QA accuracy do not necessarily have a strong correlation. This is particularly concerning here, as highly similar phrases from completely unrelated contexts might be retrieved simultaneously, potentially misleading the downstream generator. Since this paper primarily optimizes the retrieval stage, the multi-view setup might be susceptible to benchmark hacking, artificially inflating the QA scores (which ties back to the lack of rigorous multi-view discussion mentioned in Point 1). However, the experiments in Table 7, which are intended to demonstrate genuine improvements in retrieval precision, are somewhat insufficient to definitively rule out this possibility or provide a robust analysis of retrieval quality.
We appreciate the reviewer's concern that improved downstream QA accuracy does not necessarily establish better retrieval quality, and that domain-specific view induction could introduce evaluation-dependent bias. We address these concerns by clarifying the view-induction protocol and providing retrieval-only and controlled distractor evaluations.
(1) Domain-conditioned view induction versus benchmark-specific optimization.
MVP-Mem uses high-level benchmark descriptions to define the types of evidence that the memory builder should preserve. These descriptions characterize the task domain, such as everyday activities, object locations, temporal relations, and open-domain video understanding, rather than the content of individual evaluation questions.
Crucially, the semantic views are induced once from these descriptions, fixed before clip memory construction, and applied uniformly to all clips within each benchmark. Individual test questions, answer choices, answer distributions, and ground-truth annotations are never provided during view induction. All generated views are retained without manual filtering or evaluation-based selection.
This protocol distinguishes domain-conditioned memory construction from optimization using evaluation-specific questions or answers. We acknowledge that the current method employs benchmark-level domain conditioning; transferring a single fixed schema across different domains is not evaluated and remains an important direction for future work. Section 2.3 and Appendix E.1 (Tables 19, 21, and 23) now explicitly document the induction procedure and the information provided to the cloud LLM.
(2) Retrieval quality independent of downstream QA.
To assess whether the improvements reflect better evidence retrieval rather than answer-generation effects, we report retrieval-only NDCG@20 in Table 8. With the same Qwen3-VL-4B memory builder, MVP-Mem improves NDCG@20 over caption memory from 3.7 to 15.9 on EgoLifeQA, 2.3 to 22.9 on Ego-R1, and 17.1 to 22.8 on SuperMemory-VQA.
These improvements are measured independently of the answer generator and directly demonstrate better ranking of ground-truth clips. They therefore provide evidence for the retrieval contribution beyond end-to-end QA accuracy.
Author's Response to Reviewer 17Gb (1/5)
We sincerely thank the reviewer 17Gb for their insightful and constructive comments. We have carefully considered the reviewer's concerns and provide our detailed responses below.
Revision note. The manuscript revisions and additional experiments for this rebuttal are highlighted in blue.
Thank you for suggesting that we make the joint role of multi-view construction and phrase-level memory clearer. MVP-Mem's contribution is the joint design of view-conditioned memory writing and phrase-level evidence, rather than the maximum-similarity operator. Complementary semantic views decompose memory writing into focused extraction tasks, while compact phrases preserve their outputs as separately retrievable units. Treating the method as maximum scoring over generic text overlooks this construction design. We have reorganized Section 1 and Sections 2.2–2.4 to make this contribution explicit.
A memory builder must preserve potentially useful evidence without knowing future queries. Generic captioning asks the builder to identify salient observations, cover heterogeneous evidence, and integrate these observations into fluent narration within a single generation task. Our approach changes this task formulation rather than simply assigning the same captioning task to a smaller model. Multi-view construction narrows each extraction task to a specified evidence type, while the views collectively target complementary information, including actions, object states and locations, temporal cues, and object interactions. The purpose is therefore not merely to generate more content, but to make the extraction requirements explicit and reduce reliance on a single open-ended description.
Phrase-level output complements this decomposition. Once an extraction task is focused on a particular evidence type, the builder need not turn each observation into a complete sentence or combine heterogeneous observations into a coherent narrative. Instead, it produces compact phrases that retain local relational information beyond isolated object labels. Multi-view construction simplifies what each extraction task asks the model to identify; phrase-level output simplifies how the extracted evidence must be expressed. These are two coordinated choices for adapting memory writing to a lightweight builder, rather than independent additions to a generic captioning pipeline.
The resulting memory also supports selective access. Different phrases may describe different aspects of a clip, so aggregating them into one vector can obscure the evidence relevant to a particular query. We therefore mean-pool token representations within each phrase and retain separate phrase vectors for query-dependent scoring. Pooling within phrases preserves the chosen evidence unit, while keeping phrases separate allows retrieval to select among the collected observations. Maximum-similarity scoring serves this access requirement; it is not the source of our novelty claim. Section 2.4 now explicitly acknowledges ColBERT and ColBERTv2 and states that we do not claim novelty for the operator, retrieval backbone, or training objective; Section 4 provides the corresponding late-interaction context.