acceptodds
Under review as a conference paper at ICLR 2027

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

Abstract

Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive. Memory systems offer a scalable alternative by compacting videos into text representations, but often fail on practical benchmarks: either the memory does not preserve key evidence, or the retriever fails to locate relevant entries due to retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time video access, MemLife improves over the strongest training-free baseline by 4.6–12.0% across four long-horizon benchmarks. To further improve memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the memory writer to produce faithful, informative, and retrievable memories. MemOpt consistently improves MemLife by 2.7–5.0% across different video and question distributions, with gains that generalize across writer and reader backbones and memory systems.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Agent, Long-term Memory, VLM Post-training, Egocentric Video

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Ysd5OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MEMLIFE, a multimodal memory system for long-term egocentric video question answering, and MEMOPT, a reinforcement learning framework for optimizing memory writing. MEMLIFE constructs entity-grounded, first-person textual memories from video and audio, and retrieves relevant evidence through a time-indexed agentic reader. MEMOPT introduces the FIRM reward, which jointly optimizes Faithfulness, Informativeness, and Retrievability to improve memory quality. The proposed approach demonstrates strong performance across four long-term video QA benchmarks, with additional ablation and generalization experiments supporting its effectiveness.

Critical strengths
  • Clear motivation and well-designed memory framework. The paper addresses an important problem in long-term video understanding: how to construct compact memories that preserve answer-relevant information while remaining faithful to the source video and easy to retrieve. The motivation and problem formulation are clear, and the design choices of MEMLIFE are well justified.

  • Effective writer optimization and comprehensive experimental validation. MEMOPT introduces the FIRM reward to optimize memory writing in terms of faithfulness, informativeness, and retrievability. The proposed method consistently improves performance across four benchmarks, including the extended SuperMemory-LVQA and EgoLife-EQA settings. Extensive ablation and generalization experiments across different writer–reader configurations and memory systems further support the effectiveness of the proposed framework.

Critical weaknesses
  • Limited analysis of agentic retrieval behavior. While Section 3.2 and Appendix L describe the reader's retrieval tools and their implementation, the paper provides limited analysis of how the agentic reader operates across multiple rounds in practice. In particular, the average number of retrieval rounds and the effect of MEMOPT on retrieval behavior are not reported. Such an analysis would help clarify whether the optimized memories improve not only retrieval effectiveness but also retrieval efficiency.

  • Limited analysis of training data selection. MEMOPT is trained on a fixed subset of SuperMemory-VQA subjects (S1–S6). However, the original dataset contains an uneven number of questions across subjects. Although the authors demonstrate generalization to unseen subjects and benchmarks, it remains unclear how sensitive the performance improvements are to the training subject selection. An analysis using alternative subject splits or more balanced training subsets would strengthen the robustness claims.

Questions for authors
  • Agentic retrieval behavior and efficiency. The proposed reader operates through multiple retrieval rounds, with a maximum of 10 rounds (Appendix B.1). What is the average number of rounds required to answer a question? Additionally, does MEMOPT reduce the number of retrieval rounds or change the frequency of SearchMemory and FetchMemory calls? Providing representative multi-round retrieval trajectories would help clarify how the reader operates in practice.

  • Training data distribution and subject selection. Given the uneven number of questions across subjects in SuperMemory-VQA, have the authors investigated whether MEMOPT achieves consistent improvements with more balanced subject sampling or alternative training subject selections? Such an analysis would help clarify the sensitivity of writer optimization to the training data composition.

  • Category-wise performance analysis. While Figure 3 provides category-wise results on EgoLifeQA, a similar comparison between MEMLIFE and MEMLIFE + MEMOPT on SuperMemory-VQA would be valuable. In particular, reporting performance across the six question categories, along with brief descriptions of each category, would help clarify which types of questions benefit most from the proposed writer optimization.

  • Qualitative examples and failure analysis. While Table 2 provides examples of memory corrections after MEMOPT training, additional qualitative examples of the constructed memories, including complete memory entries and representative failure cases, would help clarify how the proposed writer optimization works in practice and identify its limitations.

  • Computational cost and reproducibility. For better reproducibility, it would be helpful to report the hardware configuration, training time, and GPU-hours required for MEMOPT training.

5 comments

Thank you for the detailed and constructive feedback! We have uploaded a revised manuscript with changes highlighted in blue. Below we respond point-by-point and describe the corresponding revisions. Table, figure, and appendix numbers in our responses refer to the revised manuscript.

W1. Limited analysis of agentic retrieval behavior. [...] In particular, the average number of retrieval rounds and the effect of MEMOPT on retrieval behavior are not reported. Such an analysis would help clarify whether the optimized memories improve not only retrieval effectiveness but also retrieval efficiency.

Q1. Agentic retrieval behavior and efficiency. [...] What is the average number of rounds required to answer a question? Additionally, does MEMOPT reduce the number of retrieval rounds or change the frequency of SearchMemory and FetchMemory calls? Providing representative multi-round retrieval trajectories would help clarify how the reader operates in practice.

Following your suggestion, we report the average number of rounds and tool calls per question with and without MemOpt. Each round consists of either one tool call or the final answer, so a two-round trajectory contains one retrieval call followed by the answer.

BenchmarkMemoryAccuracyRecallRounds / qSearchMemory / qFetchMemory / q
SuperMemory-VQAMemLife56.5080.533.392.390.008
+ MemOpt60.3581.493.322.320.002
EgoLifeQAMemLife52.8048.403.322.300.014
+ MemOpt56.6050.602.991.980.008

As shown in the table, the reader uses about three rounds per question, although ten rounds are available. Most retrieval calls are SearchMemory, which already supports temporal constraints through its optional time interval. On EgoLifeQA, about 9–11% of searches use this interval, for example by resolving "today" in a question to the corresponding day. FetchMemory is used only occasionally to inspect consecutive episodes within a selected interval.

With the same reader, MemOpt improves accuracy while reducing the number of retrieval calls on both benchmarks, especially on EgoLifeQA, indicating that the optimized memories improve not only retrieval effectiveness but also retrieval efficiency.

To examine the trajectories in more detail, the table below groups questions by the number of rounds used, reporting accuracy with the number of questions in parentheses.

BenchmarkMemory2 rounds3–5 rounds6–9 rounds10 rounds
SuperMemory-VQAMemLife63.0 (281)52.5 (265)47.1 (68)44.4 (9)
+ MemOpt65.7 (309)58.2 (239)41.7 (60)60.0 (15)
EgoLifeQAMemLife59.3 (290)46.3 (136)39.6 (53)38.1 (21)
+ MemOpt61.5 (296)48.8 (160)51.4 (37)57.1 (7)

With MemOpt, more questions are answered within two rounds, and accuracy in this group improves on both benchmarks. Accuracy also increases in most other groups.

As a representative trajectory, consider the EgoLifeQA question "What does Alice not eat?" With MemOpt, the reader first searches for "Alice eat food noodles bread cream pizza", which does not surface the answer. In round 2, it refines the query to "Alice not eat refuse dislike food" and retrieves an entry at rank 29 recording Alice's statement, "Can you just give me a tiny bit because I don't eat cream." It then answers correctly in round 3. With the untrained memory, five searches fail to retrieve this entry, and the reader answers incorrectly in round 6. We have added the round and tool-call statistics to Tables 21 and 22 in Appendix E of our revised paper, and representative trajectories covering single-search answers, query refinement, FetchMemory, and a failure case to Appendix M.1.

W2. Limited analysis of training data selection. MEMOPT is trained on a fixed subset of SuperMemory-VQA subjects (S1–S6). However, the original dataset contains an uneven number of questions across subjects. [...] An analysis using alternative subject splits or more balanced training subsets would strengthen the robustness claims.

Q2. Training data distribution and subject selection. Given the uneven number of questions across subjects in SuperMemory-VQA, have the authors investigated whether MEMOPT achieves consistent improvements with more balanced subject sampling or alternative training subject selections? [...]

Following your suggestion, we retrained MemOpt on a subject-balanced subset of the original training subjects (S1–S6), in which each subject contributes 177 questions with verified evidence, the number available for the smallest subject (S3). The table below lists the number of training questions from each subject.

Training setS1S2S3S4S5S6Total
Original4753951771882042141,653
Subject-balanced1771771771771771771,062

The table below compares the results of the two training sets.

MethodSuperMemory-VQA (Accuracy)SuperMemory-VQA (Recall)EgoLifeQA (Accuracy)EgoLifeQA (Recall)
MemLife56.5080.5352.8048.40
+ MemOpt (original training set)60.3581.4956.6050.60
+ MemOpt (subject-balanced training set)59.0781.3055.4049.20

With subject-balanced training, MemOpt still improves MemLife on both benchmarks and retains most of its gains, despite using 36% fewer training questions. This indicates that the benefit of writer optimization does not rely on the uneven subject distribution of the original training set. We have added the subset construction and results to Table 17 in Appendix D.

Q3. Category-wise performance analysis. While Figure 3 provides category-wise results on EgoLifeQA, a similar comparison between MEMLIFE and MEMLIFE + MEMOPT on SuperMemory-VQA would be valuable. In particular, reporting performance across the six question categories, along with brief descriptions of each category, would help clarify which types of questions benefit most [...]

Following your suggestion, we report category-wise results on SuperMemory-VQA, with EgoRAG included for reference. As defined in the SuperMemory-VQA paper, the six categories cover spoken facts and commitments from past conversations (Conversational Memory), stated or implied goals and intended future actions (Intent Recall), prior facts that must be combined with the current context (In-Context Retrieval), the chronological order of events (Timeline Reconstruction), the last known position of objects and their movement across locations (Object-Location Memory), and visual details such as text, screens, and scene contents (Visual Recall). As in Figure 3, the table below reports accuracy, retrieval recall, and oracle accuracy, which gives the reader the evidence entries directly and thus bypasses retrieval.

MetricMethodConversational MemoryIntent RecallIn-Context RetrievalTimeline ReconstructionObject-Location MemoryVisual Recall
AccuracyEgoRAG69.1170.9731.8245.3740.3734.31
MemLife63.4177.4243.1857.4152.2944.12
MemLife + MemOpt79.6788.1745.4556.4850.4639.22
RecallEgoRAG83.6759.6872.4175.5148.2856.52
MemLife71.4375.8187.3687.7675.8683.70
MemLife + MemOpt72.4579.0383.9189.8086.2177.17
Oracle AccuracyEgoRAG65.8586.0228.4152.7840.3732.35
MemLife73.9889.2538.6459.2647.7138.24
MemLife + MemOpt80.4990.3236.3658.3347.7137.25

MemLife + MemOpt outperforms EgoRAG in all six categories. Relative to MemLife, MemOpt brings the largest accuracy gains on Conversational Memory and Intent Recall. Its gain on Conversational Memory mainly reflects better memory content, as oracle accuracy rises with similar recall, whereas its gain on Intent Recall mainly reflects better retrieval, as recall rises with similar oracle accuracy. The largest decrease from MemOpt is on Visual Recall, which receives far fewer question–clip pairs during training than Conversational Memory and Intent Recall (236, compared with 602 and 469). We have added these results to Section 4.3 (Table 2), and the number of training questions and question–clip pairs per category to Appendix B.2 (Table 13).

Q4. Qualitative examples and failure analysis. [...] additional qualitative examples of the constructed memories, including complete memory entries and representative failure cases, would help clarify how the proposed writer optimization works in practice and identify its limitations.

The table below shows a success and a failure of MemOpt, with the reader's answer after each memory.

Question (correct answer)Untrained memoryMemOpt memory
"What did C say earlier about exactly what sensors the AI glasses use?" (audio, video, eye gaze, and head orientation)"someone asks if the recording includes audio and video, and I respond that it also records eye gaze and head orientation" → "cannot be answered""[A] asks, 'So, like, it is recording the audio and video, right?' and [C] responds, 'Audio, video, your eye gaze, and also your head orientation.'" → correct
"What brand was the microwave?" (Frigidaire)"stainless steel appliances including a Frigidaire microwave above the stove" → correct"a stainless steel microwave and stove" → "cannot be answered"

In both cases, the evidence clip is retrieved at a similar rank under both memories, so the difference comes from the content preserved in the entry. In the first case, the untrained writer attributes C's statement to the wearer, so the reader finds no record of C, whereas MemOpt keeps the speaker label from the transcript. This matches the largest category gain of MemOpt, on Conversational Memory (Q3). The second case, a Visual Recall question, shows that optimization can omit a visual detail needed by a later question. This is consistent with Q3, where Visual Recall shows the largest decrease and receives the fewest training question–clip pairs. We have added the complete entries of the first clip, together with its transcript, to Appendix M.2, and the retrieval trajectory of the second case to Appendix M.1.

Q5. Computational cost and reproducibility. For better reproducibility, it would be helpful to report the hardware configuration, training time, and GPU-hours required for MEMOPT training.

We train the Qwen3.5-9B writer on two nodes with eight H100 80GB GPUs each. One node runs GRPO with verl and vLLM rollouts, and the other serves the frozen Qwen3.5-9B evaluators as four replicas of two GPUs each. Each 45-step epoch takes about 6 hours, and a full three-epoch run, including checkpoint selection, takes about 25 hours and about 360 H100 GPU-hours, as some steps run on a single node. The one-time preparation before training takes about 20 additional GPU-hours. We have added these details to Appendix B.2.

Reviewer_mePGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MEMLIFE, a multimodal memory system designed to address video memory compression. Specifically, it constructs entity-grounded, first-person text episodes and retrieves memories through a time-indexed agentic reader. To further improve memory quality, the paper introduces MEMOPT, a reinforcement learning framework for generating higher-quality memories. Experiments demonstrate the effectiveness of MEMLIFE and MEMOPT.

Critical strengths

Using MEMOPT to improve the faithfulness, informativeness, and retrievability of model-generated memories is well motivated. The ablation studies demonstrate the effects of the individual reward functions, and the paper also evaluates generalization to other agents.

Critical weaknesses
  1. The comparisons are insufficient. Some baseline components are replaced with a common backbone, and the comparisons do not use the best-performing configurations of the other models. Can the reproduced baselines with substituted models match the performance of their original recommended configurations? In addition, the paper does not compare the number of input tokens, making the comparisons less convincing.
  2. MEMLIFE has limited novelty. Designs such as converting videos into textual memories, multi-round planning, and entity grounding have already been adopted by previous frameworks.
Questions for authors
  1. Regarding Weakness 1, please add comparisons with the best results of the other models and compare the number of input tokens.
  2. Regarding Weakness 2, please clarify the novelty of the MEMLIFE framework relative to other agent frameworks.
9 comments

Thank you for the detailed and constructive feedback! We have uploaded a revised manuscript with changes highlighted in blue. Below we respond point-by-point and describe the corresponding revisions. Table, figure, and appendix numbers in our responses refer to the revised manuscript.

W1. The comparisons are insufficient. Some baseline components are replaced with a common backbone, and the comparisons do not use the best-performing configurations of the other models. Can the reproduced baselines with substituted models match the performance of their original recommended configurations? [...]

Q1. Regarding Weakness 1, please add comparisons with the best results of the other models [...]

Thank you for raising this point. We would like to first clarify our evaluation protocol. For the training-free baselines, we follow their official recommended configurations and use Qwen3.5-9B, the same backbone as MemLife, for all replaceable model components, so that the comparison reflects differences in memory design rather than in backbone capacity. For the trained baselines VST, TaskMem, and M3-Agent, our revised paper reports both their released checkpoints with the original backbones and our reproductions with Qwen3.5-9B, trained on the same SuperMemory-VQA split as MemOpt (Appendix C, Table 14). The table below compares the two versions with MemLife + MemOpt.

MethodBackboneSuperMemory-VQA (Accuracy)SuperMemory-VQA (Recall)EgoLifeQA (Accuracy)EgoLifeQA (Recall)
VST (released)Qwen2.5-VL-32B37.5656.8733.603.40
VST (reproduced)Qwen3.5-9B56.5065.0829.407.60
TaskMem (released)Qwen3-VL-30B-A3B50.8865.0846.2040.20
TaskMem (reproduced)Qwen3.5-9B48.9651.1548.2035.00
M3-Agent (released)Qwen2.5-Omni-7B + Qwen3-32B43.1831.8730.407.80
M3-Agent (reproduced)Qwen3.5-9B53.9354.2035.409.60
MemLife + MemOptQwen3.5-9B60.3581.4956.6050.60

As shown in the table, MemLife + MemOpt outperforms both versions of all three methods in accuracy and recall on both benchmarks. Regarding whether the reproductions match the original configurations, neither version consistently outperforms the other across the three methods. Despite their smaller backbones, our reproductions outperform the released checkpoints of VST and M3-Agent on SuperMemory-VQA and of TaskMem on EgoLifeQA, indicating that our reproduction does not systematically disadvantage these baselines. In addition, the EgoLifeQA accuracy we obtained with the released TaskMem checkpoint (46.20) is close to the 45.4 reported in its paper [1], supporting the validity of our evaluation setup. We have clarified this evaluation protocol in Section 4.1 and Appendix A.2 of our revised paper and added the released checkpoints of all three methods to Table 14.

[1] Task-Focused Memorization for Multimodal Agents. arXiv. 2026.

W1. [...] In addition, the paper does not compare the number of input tokens, making the comparisons less convincing.

Q1. [...] and compare the number of input tokens.

Following your suggestion, the table below reports the mean input tokens per question for the compared methods. Total input tokens sum the input across all reader calls, including history repeated in subsequent rounds, while new input tokens count only the input added in each round.

MethodAccuracy (SuperMemory-VQA)Total Input Tokens (SuperMemory-VQA)New Input Tokens (SuperMemory-VQA)Accuracy (EgoLifeQA)Total Input Tokens (EgoLifeQA)New Input Tokens (EgoLifeQA)
Video ReCap36.2810k10k35.2053k53k
EgoRAG49.2824k24k48.2024k24k
Video-RAG43.9818k18k38.8019k19k
VideoARM39.3359k46k36.00183k118k
EGAgent43.6617k15k34.0029k26k
WorldMM42.0531k24k42.2046k37k
MemLife56.5058k24k52.8045k20k

MemLife uses more total input tokens than the single-pass EgoRAG, because its multi-round reader receives the earlier history again in each call. In terms of new input tokens, MemLife is comparable to EgoRAG on both benchmarks while achieving higher accuracy.

To examine whether this advantage depends on a large retrieval budget, we further reduced the number of memories returned per search from k = 32 to k = 8. The table below compares the two settings.

MethodAccuracy (SuperMemory-VQA)Total Input Tokens (SuperMemory-VQA)New Input Tokens (SuperMemory-VQA)Accuracy (EgoLifeQA)Total Input Tokens (EgoLifeQA)New Input Tokens (EgoLifeQA)
MemLife (k = 32)56.5058k24k52.8045k20k
MemLife + MemOpt (k = 32)60.3570k28k56.6077k32k
MemLife (k = 8)53.9336k10k51.0037k11k
MemLife + MemOpt (k = 8)56.3428k9k52.6045k14k

At k = 8, both total and new input tokens decrease, while MemLife still outperforms all six training-free baselines in accuracy on both benchmarks. MemOpt further improves MemLife at k = 8 on both benchmarks. Notably, on SuperMemory-VQA, MemLife + MemOpt at k = 8 achieves accuracy comparable to MemLife at k = 32 with less than half of the total input tokens. We have added these token measurements and the retrieval-budget analysis to Table 19 in Appendix E.

W2. MEMLIFE has limited novelty. Designs such as converting videos into textual memories, multi-round planning, and entity grounding have already been adopted by previous frameworks.

Q2. Regarding Weakness 2, please clarify the novelty of the MEMLIFE framework relative to other agent frameworks.

We would like to clarify the scope of our contributions. Our work studies what a system should write into long-term egocentric video memory, first designing a strong writer and reader by hand (MemLife) and then learning a better writer (MemOpt). The core methodological novelty lies in MemOpt, which you found "well motivated". Rather than applying RL to improve final answers, as in prior work on memory agents, MemOpt applies RL only to memory writing and directly rewards memories that are faithful, informative, and retrievable. With the same supervision, it outperforms RL with final-answer rewards on SuperMemory-VQA (60.35 vs. 57.46) and EgoLifeQA (56.60 vs. 52.60), without relying on stronger models (Table 7).

For MemLife, we agree that its individual components, such as textual memories, multi-round retrieval, and entity grounding, have been used in prior frameworks. Its contribution lies in identifying which designs matter for long-term egocentric memory and combining them in one system. As shown in the table below, each of the evaluated training-free systems lacks at least two of these designs, and MemLife is the only one that combines all six.

MethodMultimodal FusionEntity GroundingFirst-person NarrationAgentic ReasoningTime AnchoringChronological Ordering
Video ReCap✗✗✗✗✗✗
EgoRAG✓✗✓✗✓✗
Video-RAG✗✗✗✗✗✓
VideoARM✗✗✗✓✓✗
EGAgent✓✓✗✓✓✗
WorldMM✓✗✗✓✓✗
MemLife✓✓✓✓✓✓

The ablations in Table 6 show that each design contributes. For example, adding entity grounding and first-person narration to the multimodal writer raises SuperMemory-VQA accuracy from 52.33 to 56.50, and adding chronological ordering to the reader raises EgoLifeQA accuracy from 50.60 to 52.80 with the untrained writer. Together, these designs allow MemLife to outperform the strongest training-free baseline on each benchmark by 4.6–12.0 points without any training (Table 1). We have added this design comparison as Table 11 in Appendix A.2 of our revised paper.


Thank you again for your feedback, which helped us strengthen the baseline comparisons and clarify the contributions of our work. We hope our responses address your concerns and would be happy to discuss any remaining questions!

Reviewer_hW4gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses causal question answering over lifelong egocentric video, where a stream of 30-second video clips and transcripts is converted into persistent text memory before any question is asked at time τq\tau_q.

First, the authors propose MemLife, a training-free pipeline where a Qwen3.5-9B writer summarizes each clip into a short first-person text entry (combining visual frames and speaker-grounded transcripts), and an agentic reader iteratively searches and fetches text intervals to answer questions (with MemLife-V optionally viewing raw video frames).

Second, they introduce MemOpt, which freezes the reader and trains only the writer via GRPO. The reward (FIRM) combines three multiplicative terms: token-level Faithfulness (checked against a frozen clip-level editor), sequence-level Informativeness (whether the summary entails key facts from training QA pairs), and Retrievability (whether the candidate entry is retrieved when replaying cached reader queries).

Trained on SuperMemory-VQA (S1–S6), the method is evaluated on SuperMemory-VQA, out-of-domain EgoLifeQA, and two newly constructed benchmarks (SuperMemory-LVQA and EgoLife-EQA). MemLife outperforms prior baselines by 4.6–12.0 points, and MemOpt brings a further 2.7–5.0 point improvement, supported by ablations on reward terms, model sizes, and writer vs. reader training.

Critical strengths

S1. The diagnosis of the problem and the proposed solution are closely matched.

The paper identifies where answer-relevant information is lost: evidence can be dropped or distorted when the video is written into memory, and it can fail to surface when the reader searches a long, crowded history. Each part of the method then targets one of these failure points. On the writing side, multimodal fusion and entity grounding aim to preserve evidence; on the retrieval side, first-person narration and time-anchored, chronologically ordered access aim to make it findable; and the three FIRM rewards map one-to-one onto faithfulness, informativeness and retrievability. The reward ablation supports this mapping, since each signal alone fixes one failure mode while hurting another.

S2. It reframes the object of optimization from the reader’s reasoning to the memory itself.

Most prior work improves memory QA by training the reader or by end-to-end answer rewards. This paper instead treats the written memory as a persistent, inspectable artifact and asks how to supervise it directly, using signals that do not depend on the reader’s reasoning. Replaying cached reader actions to score retrievability, rather than re-running the agent, makes this practical. Because the result is better text memory rather than a better policy, the improvement can in principle be reused by other readers and systems, as the transfer to the EgoRAG pipeline suggests.

Critical weaknesses

W1. ‘Writer-only’ is not strongly supported.

With the 27B reader the gain falls to +0.97 / +0.60, versus +10.4 / +6.8 from swapping the reader itself (Table 3). Also, the reader reward includes overlap-based recall, which rewards broad fetching, and training uses 6 rounds vs. 10 at evaluation. On EgoLifeQA, reader training gives recall +10.6 but accuracy −1.4 (within std 1.83), which is consistent with recall hacking rather than “specialization to the training distribution”. Recall drops after MemOpt on SuperMemory-LVQA (46.18 → 43.13) and EgoLife-EQA (44.00 → 41.00).

W2. The writer’s training signal may not be question-agnostic.

Both Informativeness and Retrievability are defined over Q(c), so training needs (question, answer, evidence clip) triples. These are unavailable for real lifelogs and are strictly more supervision than final-answer RL, so Table 5 is not supervision-matched. This is not discussed, and there is no annotation-budget study. Also, only annotated, mostly eventful clips are optimized; no term penalizes irrelevant content; and clips reached by fetch always receive R_ret = 1. Retrievability optimizes similarity to queries from first-person training questions, and every benchmark asks in the first person, yet robustness to question phrasing (e.g., third-person or neutral rewrites of the test questions) is not evaluated.

Questions for authors

Q1. Robustness to question phrasing: rewrite the test questions in third-person or neutral format.

Q2. Significance and a controlled comparison: multi-seed results with significance tests for MemOpt vs. MemLife, and a reader trained with an accuracy-only reward and a 10-round budget. Do the gains and the EgoLifeQA conclusion of App. H still hold?

Q3. How are clips with an empty Q(c) handled during MemOpt training? What fraction of training clips is actually optimized, and how does average memory length change before and after MemOpt?

Q4. The key-fact extractor and faithfulness evaluator are the same frozen Qwen3.5-9B as the writer and see the same eight frames. How can the evaluator catch perception errors that the writer itself makes? (Eq. 7) aligns with human-annotated hallucination spans. Under prefix teacher forcing, an unsupported multi-token span or a deletion edit may be penalized only at its first token.

5 comments

Thank you for the detailed and constructive feedback! We have uploaded a revised manuscript with changes highlighted in blue. Below we respond point-by-point and describe the corresponding revisions. Table, figure, and appendix numbers in our responses refer to the revised manuscript.

W1. 'Writer-only' is not strongly supported. With the 27B reader the gain falls to +0.97 / +0.60, versus +10.4 / +6.8 from swapping the reader itself (Table 3) [...]

Q2. Significance and a controlled comparison: multi-seed results with significance tests for MemOpt vs. MemLife [...]

We agree that the gain of MemOpt is smaller with the 27B reader. To test its significance, we repeat question answering with the 27B reader over five seeds at temperature 1.0. The table below reports accuracy as mean ± standard deviation across seeds, with 95% confidence intervals and p-values from paired comparisons at the question level.

Condition (27B reader)SuperMemory-VQAEgoLifeQA
Zero-shot memory66.65 ± 0.8457.40 ± 1.39
MemOpt memory65.14 ± 1.0561.56 ± 2.48
Oracle context68.22 ± 0.4968.68 ± 0.95
MemOpt − zero-shot (95% CI, p)−1.51 ([−3.95, +0.87], p = 0.24)+4.16 ([+1.36, +6.96], p = 0.004)
Oracle − zero-shot (95% CI, p)+1.57 ([−1.96, +5.04], p = 0.39)+11.28 ([+7.00, +15.60], p < 0.001)

As shown in the table, the gain of MemOpt is larger when the reader has more room for improvement. On EgoLifeQA, where zero-shot memory leaves the 27B reader far below oracle context, MemOpt significantly improves accuracy and closes 37% of this gap, showing that the optimized memory also benefits a reader that was not used to collect retrievability supervision. On SuperMemory-VQA, zero-shot memory already brings the 27B reader close to oracle context, leaving little room for any memory to help. With the 9B reader, which leaves more headroom, MemOpt improves accuracy by 2.7–5.0 points on all four benchmarks (Table 1).

We also view reader scaling and writer optimization as complementary rather than competing, since the best results in Table 4 (Table 3 in the original submission) combine the 27B reader with a MemOpt-trained writer. Moreover, scaling the reader triples the model size used to answer every question, whereas MemOpt is trained once and leaves the reader unchanged. As you noted in S2, because the result is better memory rather than a better policy, "the improvement can in principle be reused by other readers and systems", as supported by Table 4 and Figure 4. We have added these multi-seed results (Table 16) and the statistical procedure to Appendix D of our revised paper, and revised Section 4.4 to discuss how the gain of writer optimization depends on the reader's headroom.

W1. [...] the reader reward includes overlap-based recall, which rewards broad fetching, and training uses 6 rounds vs. 10 at evaluation. On EgoLifeQA, reader training gives recall +10.6 but accuracy −1.4 (within std 1.83), which is consistent with recall hacking rather than "specialization to the training distribution" [...]

Q2. [...] and a reader trained with an accuracy-only reward and a 10-round budget. Do the gains and the EgoLifeQA conclusion of App. H still hold?

Following your suggestion, we trained a reader with an accuracy-only reward, keeping the six-round training budget and all other settings of Appendix H. The conclusion of Appendix H still holds, and the new result indicates that it is not explained by recall hacking. The table below compares the two reader rewards with writer-only training, where Rounds / q counts all reader calls per question, including the final answer.

TrainingSuperMemory-VQA (Accuracy)SuperMemory-VQA (Recall)SuperMemory-VQA (Rounds / q)EgoLifeQA (Accuracy)EgoLifeQA (Recall)EgoLifeQA (Rounds / q)
None56.5080.533.3952.8048.403.32
Reader RL: (accuracy + recall) / 261.0089.504.7951.4059.003.83
Reader RL: accuracy only66.1373.662.0049.8042.802.00
Writer RL: MemOpt60.3581.493.3256.6050.602.99

The recall term can indeed encourage broader retrieval, as the reader trained with (accuracy + recall) / 2 uses more rounds than the untrained reader and attains higher recall. However, removing the recall term does not restore accuracy on EgoLifeQA. The accuracy-only reader learns to answer after a single search, which raises in-domain accuracy but further lowers EgoLifeQA accuracy. One possible explanation is that a single search often suffices on the short SuperMemory-VQA histories but not on the much longer EgoLifeQA history (3.97 vs. 22.65 video hours on average, Table 10), where this reader's recall also drops. Under both rewards, reader training improves in-domain accuracy without transferring to EgoLifeQA, whereas writer-only MemOpt improves EgoLifeQA accuracy with the same untrained reader.

Both readers are trained with a six-round budget to reduce training cost, and this budget is rarely restrictive. With ten rounds available at evaluation, the untrained reader finishes within six rounds on 93.9% of SuperMemory-VQA and 89.8% of EgoLifeQA questions, and both trained readers use fewer than five rounds on average. Appendix H also assigns a reward of −1 to trajectories that exhaust the training budget, which we introduced because the reader trained with the combined reward but without this penalty always used all ten rounds at evaluation. We have added the accuracy-only reader (Table 25), the round statistics, and the motivation for this penalty to Appendix H.

W1. [...] Recall drops after MemOpt on SuperMemory-LVQA (46.18 → 43.13) and EgoLife-EQA (44.00 → 41.00).

While MemOpt does not increase annotated recall on these two extended settings, it still improves accuracy (56.18 → 58.91 on SuperMemory-LVQA and 52.00 → 57.00 on EgoLife-EQA). As discussed in Section 4.2, this indicates that the gains also come from more answer-useful memory content, not only from retrieving annotated evidence. Figure 3 illustrates this on EgoLifeQA, where MemOpt slightly lowers recall on EventRecall and EntityLog questions but raises their oracle accuracy, which gives the reader the evidence entries directly and thus measures how useful the entries are once retrieved. This improvement in the entries outweighs the small loss in recall, so accuracy on both categories still rises.

W2. The writer's training signal may not be question-agnostic. Both Informativeness and Retrievability are defined over Q(c), so training needs (question, answer, evidence clip) triples. These are unavailable for real lifelogs [...]

Thank you for raising this point. We agree that the training signal of the writer is question-dependent. In our paper, "question-independent" refers to memory construction, in which the writer writes each entry from the frames and transcript alone, without seeing any question, during both training and deployment. Questions and answers are used only to compute the training rewards.

These triples are needed only for training. The writer trained on subjects S1–S6 improves accuracy on the held-out subjects S9–S10 and on EgoLifeQA and EgoLife-EQA, whose videos and questions are never used in training (Table 1). Adapting MemOpt to a new domain does require evidence-linked examples, which can be obtained by generating questions from recorded lifelogs, as done to construct SuperMemory-VQA. We have made this distinction explicit in Section 3.3 and discussed the need for such triples in Appendix B.2.

W2. [...] and are strictly more supervision than final-answer RL, so Table 5 is not supervision-matched. This is not discussed, and there is no annotation-budget study [...]

We would like to clarify that the comparison in Table 7 (Table 5 in the original submission) is supervision-matched. The clip-centric final-answer RL baseline uses the same (question, answer, evidence clip) triples as MemOpt, taking the final-answer accuracy on each clip's associated questions as the writer reward, and MemOpt's key facts are extracted from the same triples by the frozen 9B extractor without additional human annotation. With the same supervision, MemOpt outperforms this baseline on both SuperMemory-VQA (60.35 vs. 57.46) and EgoLifeQA (56.60 vs. 52.60). We also tested a question-centric variant of final-answer RL, which optimizes the memories retrieved for each question and thus requires no evidence-clip annotations. On SuperMemory-VQA, it achieves 56.82, below both the clip-centric variant and MemOpt.

MemOpt also remains effective with fewer annotations. Trained on a subject-balanced subset with 36% fewer training questions (1,062 instead of 1,653), it still improves MemLife by 2.57 points on SuperMemory-VQA and 2.60 points on EgoLifeQA, retaining most of the gains obtained with all questions. We have specified the supervision used by each method in the caption of Table 7 and added this analysis to Table 17 in Appendix D.

W2. [...] Also, only annotated, mostly eventful clips are optimized; no term penalizes irrelevant content; and clips reached by fetch always receive R_ret = 1 [...]

Q3. How are clips with an empty Q(c) handled during MemOpt training? What fraction of training clips is actually optimized, and how does average memory length change before and after MemOpt?

Clips with an empty Q(c) are not sampled for writer RL. They remain in the memory bank, which is rebuilt with the current writer after each epoch (Appendix B.3), so the retrievability of each candidate is measured against the full memory bank, including unannotated entries. In total, 1,451 of the 4,445 training clips (32.6%) have associated questions and are optimized directly. The table below reports the average memory length before and after MemOpt, under the same 512-token generation cap (Appendix B.1).

Test Data#MemoriesAvg. Tokens (Zero-shot)Avg. Tokens (MemOpt)
SuperMemory-VQA752245.5280.6
EgoLifeQA6,250237.7370.6

MemOpt increases the average memory length by 14.3% on SuperMemory-VQA and 55.9% on EgoLifeQA. During training, the average response length rises in the early steps, decreases in the middle of training, and rises again later, while remaining well below the 512-token cap (Figure 5). This non-monotonic trend suggests that the rewards do not simply favor longer memories.

We deliberately do not penalize content unrelated to the training questions, because supported details may be needed by future questions, and such a penalty would encourage the writer to drop them from persistent memory. At the same time, faithfulness penalizes unsupported content, and the KL term keeps the writer close to its initialization (Appendix B.2). Moreover, although only annotated clips are optimized, the trained writer writes every entry at test time, including clips unrelated to any question, and accuracy still improves on all four benchmarks (Table 1).

For clips reached by FetchMemory, the entry is retrieved by its timestamp, so its wording cannot change whether it is fetched. We therefore credit a fetched entry as retrieved (R_ret = 1), as for entries returned by search, and these candidates are still distinguished by the faithfulness and informativeness rewards. This case is also uncommon, as the reader rarely calls FetchMemory (0.008 calls per question on SuperMemory-VQA, compared with 2.39 SearchMemory calls, Table 21). We have added the memory-length statistics to Table 20 in Appendix E and the response length during training to Figure 5 in Appendix G, and clarified the treatment of fetched clips in Section 3.3.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your time and effort in evaluating our paper. We are grateful for the positive feedback from all reviewers, and for their constructive comments that have helped us further strengthen our submission. As the author-reviewer discussion phase is ending, we would like to summarize the key contributions highlighted by the reviewers and the revisions we have made to address their concerns. We hope this provides a clear picture of the submission after rebuttal and facilitates your subsequent discussions.

Key Contributions

  • Memory-centric problem formulation: We address the important problem of what to write into long-term video memory (Ysd5), reframing the optimization target from the QA model to the memory itself (hW4g) based on a clear diagnosis of information loss (hW4g, Ysd5).
  • Well-motivated and effective method: MemLife's well-justified design choices directly address the diagnosed failure points (hW4g, Ysd5), and MemOpt trains the memory writer with well-motivated and effective rewards for faithful, informative, and retrievable memories (mePG, Ysd5), each of which is supported by ablations (hW4g, mePG, Ysd5).
  • Consistent and transferable gains: Without training, MemLife outperforms all baselines, and MemOpt further improves it consistently across benchmarks (Ysd5), with gains that generalize to unseen data and transfer to other memory systems (hW4g, mePG, Ysd5).

Key Revisions

  • Stronger baseline comparisons: For the trained baselines, we included their released checkpoints with the original backbones alongside our reproductions, and MemLife + MemOpt outperforms both versions of the compared methods (Table 14). We also evaluated Gemma4-31B and Gemini3.5-Flash as additional backbones from other model families, confirming that MemLife's advantages generalize across models (Table 11(c)).
  • Robustness of the gains: Multi-seed results confirm that MemOpt-trained memory still brings significant improvements for the larger 27B reader when memory quality is the bottleneck (Table 16). Also, MemOpt remains effective with 36% fewer training questions, preserving most of its gains (Table 17). When the test questions are no longer phrased in the first person, MemLife keeps its advantage over EgoRAG and MemOpt continues to improve it (Table 5).
  • Additional analyses: We added analyses of retrieval behavior (Tables 21 and 22), per-category performance on SuperMemory-VQA (Table 2), input-token usage (Table 19), and the computational cost of MemOpt training (Appendix B.2), along with a human evaluation of the faithfulness evaluator (Appendix L) and qualitative examples (Appendix M).

We believe these revisions have further improved the quality of our paper and hope they will strengthen your confidence in our submission.

Thank you again for your time and consideration.

Sincerely,

Authors of Submission 7622

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.