- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MEMLIFE, a multimodal memory system for long-term egocentric video question answering, and MEMOPT, a reinforcement learning framework for optimizing memory writing. MEMLIFE constructs entity-grounded, first-person textual memories from video and audio, and retrieves relevant evidence through a time-indexed agentic reader. MEMOPT introduces the FIRM reward, which jointly optimizes Faithfulness, Informativeness, and Retrievability to improve memory quality. The proposed approach demonstrates strong performance across four long-term video QA benchmarks, with additional ablation and generalization experiments supporting its effectiveness.
Critical strengths
-
Clear motivation and well-designed memory framework. The paper addresses an important problem in long-term video understanding: how to construct compact memories that preserve answer-relevant information while remaining faithful to the source video and easy to retrieve. The motivation and problem formulation are clear, and the design choices of MEMLIFE are well justified.
-
Effective writer optimization and comprehensive experimental validation. MEMOPT introduces the FIRM reward to optimize memory writing in terms of faithfulness, informativeness, and retrievability. The proposed method consistently improves performance across four benchmarks, including the extended SuperMemory-LVQA and EgoLife-EQA settings. Extensive ablation and generalization experiments across different writer–reader configurations and memory systems further support the effectiveness of the proposed framework.
Critical weaknesses
-
Limited analysis of agentic retrieval behavior. While Section 3.2 and Appendix L describe the reader's retrieval tools and their implementation, the paper provides limited analysis of how the agentic reader operates across multiple rounds in practice. In particular, the average number of retrieval rounds and the effect of MEMOPT on retrieval behavior are not reported. Such an analysis would help clarify whether the optimized memories improve not only retrieval effectiveness but also retrieval efficiency.
-
Limited analysis of training data selection. MEMOPT is trained on a fixed subset of SuperMemory-VQA subjects (S1–S6). However, the original dataset contains an uneven number of questions across subjects. Although the authors demonstrate generalization to unseen subjects and benchmarks, it remains unclear how sensitive the performance improvements are to the training subject selection. An analysis using alternative subject splits or more balanced training subsets would strengthen the robustness claims.
Questions for authors
-
Agentic retrieval behavior and efficiency. The proposed reader operates through multiple retrieval rounds, with a maximum of 10 rounds (Appendix B.1). What is the average number of rounds required to answer a question? Additionally, does MEMOPT reduce the number of retrieval rounds or change the frequency of SearchMemory and FetchMemory calls? Providing representative multi-round retrieval trajectories would help clarify how the reader operates in practice.
-
Training data distribution and subject selection. Given the uneven number of questions across subjects in SuperMemory-VQA, have the authors investigated whether MEMOPT achieves consistent improvements with more balanced subject sampling or alternative training subject selections? Such an analysis would help clarify the sensitivity of writer optimization to the training data composition.
-
Category-wise performance analysis. While Figure 3 provides category-wise results on EgoLifeQA, a similar comparison between MEMLIFE and MEMLIFE + MEMOPT on SuperMemory-VQA would be valuable. In particular, reporting performance across the six question categories, along with brief descriptions of each category, would help clarify which types of questions benefit most from the proposed writer optimization.
-
Qualitative examples and failure analysis. While Table 2 provides examples of memory corrections after MEMOPT training, additional qualitative examples of the constructed memories, including complete memory entries and representative failure cases, would help clarify how the proposed writer optimization works in practice and identify its limitations.
-
Computational cost and reproducibility. For better reproducibility, it would be helpful to report the hardware configuration, training time, and GPU-hours required for MEMOPT training.
5 comments
W2. Limited analysis of training data selection. MEMOPT is trained on a fixed subset of SuperMemory-VQA subjects (S1–S6). However, the original dataset contains an uneven number of questions across subjects. [...] An analysis using alternative subject splits or more balanced training subsets would strengthen the robustness claims.
Q2. Training data distribution and subject selection. Given the uneven number of questions across subjects in SuperMemory-VQA, have the authors investigated whether MEMOPT achieves consistent improvements with more balanced subject sampling or alternative training subject selections? [...]
Following your suggestion, we retrained MemOpt on a subject-balanced subset of the original training subjects (S1–S6), in which each subject contributes 177 questions with verified evidence, the number available for the smallest subject (S3). The table below lists the number of training questions from each subject.
| Training set | S1 | S2 | S3 | S4 | S5 | S6 | Total |
|---|---|---|---|---|---|---|---|
| Original | 475 | 395 | 177 | 188 | 204 | 214 | 1,653 |
| Subject-balanced | 177 | 177 | 177 | 177 | 177 | 177 | 1,062 |
The table below compares the results of the two training sets.
| Method | SuperMemory-VQA (Accuracy) | SuperMemory-VQA (Recall) | EgoLifeQA (Accuracy) | EgoLifeQA (Recall) |
|---|---|---|---|---|
| MemLife | 56.50 | 80.53 | 52.80 | 48.40 |
| + MemOpt (original training set) | 60.35 | 81.49 | 56.60 | 50.60 |
| + MemOpt (subject-balanced training set) | 59.07 | 81.30 | 55.40 | 49.20 |
With subject-balanced training, MemOpt still improves MemLife on both benchmarks and retains most of its gains, despite using 36% fewer training questions. This indicates that the benefit of writer optimization does not rely on the uneven subject distribution of the original training set. We have added the subset construction and results to Table 17 in Appendix D.
Q3. Category-wise performance analysis. While Figure 3 provides category-wise results on EgoLifeQA, a similar comparison between MEMLIFE and MEMLIFE + MEMOPT on SuperMemory-VQA would be valuable. In particular, reporting performance across the six question categories, along with brief descriptions of each category, would help clarify which types of questions benefit most [...]
Following your suggestion, we report category-wise results on SuperMemory-VQA, with EgoRAG included for reference. As defined in the SuperMemory-VQA paper, the six categories cover spoken facts and commitments from past conversations (Conversational Memory), stated or implied goals and intended future actions (Intent Recall), prior facts that must be combined with the current context (In-Context Retrieval), the chronological order of events (Timeline Reconstruction), the last known position of objects and their movement across locations (Object-Location Memory), and visual details such as text, screens, and scene contents (Visual Recall). As in Figure 3, the table below reports accuracy, retrieval recall, and oracle accuracy, which gives the reader the evidence entries directly and thus bypasses retrieval.
| Metric | Method | Conversational Memory | Intent Recall | In-Context Retrieval | Timeline Reconstruction | Object-Location Memory | Visual Recall |
|---|---|---|---|---|---|---|---|
| Accuracy | EgoRAG | 69.11 | 70.97 | 31.82 | 45.37 | 40.37 | 34.31 |
| MemLife | 63.41 | 77.42 | 43.18 | 57.41 | 52.29 | 44.12 | |
| MemLife + MemOpt | 79.67 | 88.17 | 45.45 | 56.48 | 50.46 | 39.22 | |
| Recall | EgoRAG | 83.67 | 59.68 | 72.41 | 75.51 | 48.28 | 56.52 |
| MemLife | 71.43 | 75.81 | 87.36 | 87.76 | 75.86 | 83.70 | |
| MemLife + MemOpt | 72.45 | 79.03 | 83.91 | 89.80 | 86.21 | 77.17 | |
| Oracle Accuracy | EgoRAG | 65.85 | 86.02 | 28.41 | 52.78 | 40.37 | 32.35 |
| MemLife | 73.98 | 89.25 | 38.64 | 59.26 | 47.71 | 38.24 | |
| MemLife + MemOpt | 80.49 | 90.32 | 36.36 | 58.33 | 47.71 | 37.25 |
MemLife + MemOpt outperforms EgoRAG in all six categories. Relative to MemLife, MemOpt brings the largest accuracy gains on Conversational Memory and Intent Recall. Its gain on Conversational Memory mainly reflects better memory content, as oracle accuracy rises with similar recall, whereas its gain on Intent Recall mainly reflects better retrieval, as recall rises with similar oracle accuracy. The largest decrease from MemOpt is on Visual Recall, which receives far fewer question–clip pairs during training than Conversational Memory and Intent Recall (236, compared with 602 and 469). We have added these results to Section 4.3 (Table 2), and the number of training questions and question–clip pairs per category to Appendix B.2 (Table 13).
Q4. Qualitative examples and failure analysis. [...] additional qualitative examples of the constructed memories, including complete memory entries and representative failure cases, would help clarify how the proposed writer optimization works in practice and identify its limitations.
The table below shows a success and a failure of MemOpt, with the reader's answer after each memory.
| Question (correct answer) | Untrained memory | MemOpt memory |
|---|---|---|
| "What did C say earlier about exactly what sensors the AI glasses use?" (audio, video, eye gaze, and head orientation) | "someone asks if the recording includes audio and video, and I respond that it also records eye gaze and head orientation" → "cannot be answered" | "[A] asks, 'So, like, it is recording the audio and video, right?' and [C] responds, 'Audio, video, your eye gaze, and also your head orientation.'" → correct |
| "What brand was the microwave?" (Frigidaire) | "stainless steel appliances including a Frigidaire microwave above the stove" → correct | "a stainless steel microwave and stove" → "cannot be answered" |
In both cases, the evidence clip is retrieved at a similar rank under both memories, so the difference comes from the content preserved in the entry. In the first case, the untrained writer attributes C's statement to the wearer, so the reader finds no record of C, whereas MemOpt keeps the speaker label from the transcript. This matches the largest category gain of MemOpt, on Conversational Memory (Q3). The second case, a Visual Recall question, shows that optimization can omit a visual detail needed by a later question. This is consistent with Q3, where Visual Recall shows the largest decrease and receives the fewest training question–clip pairs. We have added the complete entries of the first clip, together with its transcript, to Appendix M.2, and the retrieval trajectory of the second case to Appendix M.1.
Q5. Computational cost and reproducibility. For better reproducibility, it would be helpful to report the hardware configuration, training time, and GPU-hours required for MEMOPT training.
We train the Qwen3.5-9B writer on two nodes with eight H100 80GB GPUs each. One node runs GRPO with verl and vLLM rollouts, and the other serves the frozen Qwen3.5-9B evaluators as four replicas of two GPUs each. Each 45-step epoch takes about 6 hours, and a full three-epoch run, including checkpoint selection, takes about 25 hours and about 360 H100 GPU-hours, as some steps run on a single node. The one-time preparation before training takes about 20 additional GPU-hours. We have added these details to Appendix B.2.
Thank you for the detailed and constructive feedback! We have uploaded a revised manuscript with changes highlighted in blue. Below we respond point-by-point and describe the corresponding revisions. Table, figure, and appendix numbers in our responses refer to the revised manuscript.
Following your suggestion, we report the average number of rounds and tool calls per question with and without MemOpt. Each round consists of either one tool call or the final answer, so a two-round trajectory contains one retrieval call followed by the answer.
As shown in the table, the reader uses about three rounds per question, although ten rounds are available. Most retrieval calls are SearchMemory, which already supports temporal constraints through its optional time interval. On EgoLifeQA, about 9–11% of searches use this interval, for example by resolving "today" in a question to the corresponding day. FetchMemory is used only occasionally to inspect consecutive episodes within a selected interval.
With the same reader, MemOpt improves accuracy while reducing the number of retrieval calls on both benchmarks, especially on EgoLifeQA, indicating that the optimized memories improve not only retrieval effectiveness but also retrieval efficiency.
To examine the trajectories in more detail, the table below groups questions by the number of rounds used, reporting accuracy with the number of questions in parentheses.
With MemOpt, more questions are answered within two rounds, and accuracy in this group improves on both benchmarks. Accuracy also increases in most other groups.
As a representative trajectory, consider the EgoLifeQA question "What does Alice not eat?" With MemOpt, the reader first searches for "Alice eat food noodles bread cream pizza", which does not surface the answer. In round 2, it refines the query to "Alice not eat refuse dislike food" and retrieves an entry at rank 29 recording Alice's statement, "Can you just give me a tiny bit because I don't eat cream." It then answers correctly in round 3. With the untrained memory, five searches fail to retrieve this entry, and the reader answers incorrectly in round 6. We have added the round and tool-call statistics to Tables 21 and 22 in Appendix E of our revised paper, and representative trajectories covering single-search answers, query refinement, FetchMemory, and a failure case to Appendix M.1.