acceptodds
Under review as a conference paper at ICLR 2027

The Right Memory at the Right Time: Memory-Driven Proactive Assistance

Abstract

As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce the memory-driven proactive assistance (MPA) benchmark, pairing egocentric video with year-long, distractor-rich user memories ( 618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every model we evaluate scores near zero, including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window a frontier model reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build Gated Proactive Memory (GPM), a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. GPM lifts strict accuracy from near zero to 37.0% on Qwen3.8-27B and, with no training at all, 38% on GPT-5.3-Chat; RL post-training reaches 42.8%. We will release the MPA benchmark upon publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Proactive Agent, Memory Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wvxtOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a valuable benchmark and demonstrates substantial improvements in memory-driven proactive assistance, but limited comparisons with strong baselines and unresolved evaluation validity concerns prevent a stronger recommendation.

Critical strengths

1. Well-designed benchmark and evaluation: The paper introduces a meaningful benchmark that jointly evaluates memory retrieval, intervention timing, and response correctness, addressing an important gap in proactive assistance research.

2. Strong empirical results and ablation studies: GPM substantially improves performance across different models and settings, while controlled experiments and ablations provide convincing evidence for the effectiveness of its key components.

Critical weaknesses

1. Limited comparison with strong baselines: The paper lacks direct comparisons with closely related graph-based memory systems, such as EgoMemo, making it difficult to establish GPM's incremental novelty (Section 6.1).

2. Limited benchmark realism: The benchmark relies on synthetic personal memories and predefined trigger windows, which may not fully reflect complex real-world proactive assistance scenarios (Section 3, Appendix C).

3. Inconsistent trigger-window definition: Section 3.1 and Appendix C appear to define trigger windows differently, raising concerns about the validity of timing-based evaluation.

4. Insufficient evaluation reliability: The paper relies on an LLM judge without reporting independent human agreement on model-generated responses, and lacks confidence intervals or multi-seed evaluations (Sections 3.3 and 6).

5. Limited real-world validation: Although the benchmark uses real egocentric videos, all personal histories and reminder rationales are synthetically constructed. Without evaluation on naturally collected user memories or real-world longitudinal interactions, it remains unclear whether the proposed method generalizes to authentic proactive assistance scenarios.

Questions for authors

Q1. How much of GPM's improvement remains against a strong graph-memory proactive baseline?

EgoMemo already performs memory-augmented proactive reasoning using a semantic knowledge graph and retrieved temporal context. Could the authors adapt an EgoMemo-style baseline to MPA with the same generator backbone, or provide a comparably strong structured-memory baseline with matched inference budget?

Q2. How are intermittent target appearances handled in the actual trigger-window annotations?

Section 3.1 defines the valid window using the first appearance, whereas Appendix C Step 1 describes using the first and last visible timestamps over the clip. Which rule is used by the released annotations and evaluation implementation? How many samples have multiple disjoint target appearances?

8 comments
AuthorsOpenReview ↗

Response to Reviewer wvxt (1/4)

Dear Reviewer wvxt,

We sincerely thank you for your detailed review and constructive suggestions. We hope our responses below address your concerns and merit a better score.

W1. Comparison with strong graph-memory baselines (e.g., EgoMemo).

"Limited comparison with strong baselines: The paper lacks direct comparisons with closely related graph-based memory systems, such as EgoMemo, making it difficult to establish GPM's incremental novelty (Section 6.1)."

Thank you for this great suggestion. We acknowledge that EgoServe and EgoMemo are excellent, foundational concurrent work on proactive assistance, and EgoMemo is a strong graph-memory method in this direction. Below, we first clarify the setting differences and then establish a direct comparison.

Setting differences. EgoServe studies short-horizon assistance within the same day; our MPA benchmark targets reminders that can only be justified by long-term personal memory, and evaluates them strictly.

EgoServe / EgoMemoMPA (ours)
MemoryBuilt by the system from the user's earlier video; memory-recall gaps ≤ 1 h (mostly)Year-long personal history (~618K tokens), distractor-rich
EvidenceMined by Gemini from EgoLife captionsExplicitly annotated and human-verified
ScoringA short identified trigger window plus a ±60 s tolerance on each side; content scored separatelyHuman-verified trigger window; content and timing scored jointly

Direct comparison. Following your suggestion, we compare GPM with EgoMemo in two ways: (1) in our setting, we adapt EgoMemo to our MPA benchmark and compare it with GPM; (2) in their setting, we run GPM on EgoLife data annotated by EgoServe and compare it with EgoMemo.

(1) EgoMemo on our MPA benchmark. We adapt EgoMemo with its native pipeline and our scoring rule, and evaluate on 100 randomly sampled clips. To match the generator backbone, we also run EgoMemo with GPM's decision model, Qwen3.8-27B:

MethodPERFECT (%)HIT (%)
EgoMemo (GPT-5.3 decision)4.08.0
EgoMemo (Qwen3.8-27B decision)0.08.0
GPM (ours, Qwen3.8-27B)38.055.0

(2) GPM on EgoServe's EgoLife data. Here both the egocentric video and the personal history (the user’s own earlier recordings) are real. Unlike our benchmark, which annotates the full span from when the target object appears to when it disappears, EgoServe identifies a short trigger window and adds a ±60 s scoring tolerance on each side. Since most EgoServe services provide general assistance rather than memory-driven help (e.g., safety alerts, tool-use and next-step guidance), we keep only the memory-related types: Memory Recall, Memory Link, and Task Reminder. To ensure that the memory evidence lies outside both the test clip and the ±60 s tolerance window, we further keep only clips whose evidence occurs at least 3 minutes before the trigger, yielding 65 clips. The reminder can thus only be justified by memory rather than by the current view. We first follow EgoServe's ±60 s protocol. However, ±60 s is too loose for proactive reminders: a reminder up to a minute early or late still counts as correct, so being scored correct says little about whether it came at the right moment. We therefore additionally report a tighter ±14 s window, comparable to the median trigger-window length in our benchmark (24 s).

MethodPERFECT, ±60 s (%)HIT, ±60 s (%)PERFECT, ±14 s (%)HIT, ±14 s (%)
EgoMemo12.312.37.77.7
GPM (ours)38.583.121.563.1

In both settings, GPM clearly outperforms EgoMemo.

Why EgoMemo performs poorly in our setting. EgoMemo's pipeline first looks at the current scene and then decides whether to retrieve memory. This design is well suited to cases where the object itself is worth a reminder, such as a visible hazard or an unfinished step on the spot. In memory-driven proactive assistance, however, the object in view is often unremarkable on its own; it is the user's memory that makes it worth a reminder. Without memory, it is hard to tell from the scene alone whether retrieval is needed, so EgoMemo often chooses not to retrieve at the very objects that should trigger a reminder (it issues no reminder on 86% of clips on our benchmark), which leads to its low HIT and PERFECT.

Why GPM performs well. GPM keeps the relevant memory in view and then uses its gate to post-filter what to say based on that memory. This balances both sides: it does not miss memory-driven triggers, while the gate filters out unnecessary reminders.

AuthorsOpenReview ↗

Response to Reviewer wvxt (2/4)

W2 & W5. Benchmark realism and real-world validation.

W2: "Limited benchmark realism: The benchmark relies on synthetic personal memories and predefined trigger windows, which may not fully reflect complex real-world proactive assistance scenarios (Section 3, Appendix C)."

W5: "Limited real-world validation: Without evaluation on naturally collected user memories or real-world longitudinal interactions, it remains unclear whether the proposed method generalizes to authentic proactive assistance scenarios."

We appreciate the reviewer raising these points. Both concern how our benchmark and method relate to real data, so we answer them together. We first show that our method works on real memories, and then explain how the realism of the synthetic memories and the predefined trigger windows is ensured.

Evaluation on real memories. We evaluate on EgoLife data annotated by EgoServe, where both the egocentric video and the personal history (the user's own earlier recordings) are real. Unlike our benchmark, which annotates the full span from when the target object appears to when it disappears, EgoServe identifies a short trigger window and adds a ±60 s scoring tolerance on each side. Since most EgoServe services provide general assistance rather than memory-driven help (e.g., safety alerts, tool-use and next-step guidance), we keep only the memory-related types: Memory Recall, Memory Link, and Task Reminder. To ensure that the memory evidence lies outside both the test clip and the ±60 s tolerance window, we further keep only clips whose evidence occurs at least 3 minutes before the trigger, yielding 65 clips. The reminder can thus only be justified by memory rather than by the current view. We first follow EgoServe's ±60 s protocol. However, ±60 s is too loose for proactive reminders: a reminder up to a minute early or late still counts as correct, so being scored correct says little about whether it came at the right moment. We therefore additionally report a tighter ±14 s window, comparable to the median trigger-window length in our benchmark (24s+2s tolerance). We compare GPM with EgoMemo, a graph-memory method designed for EgoLife:

MethodPERFECT, ±60 s (%)HIT, ±60 s (%)PERFECT, ±14 s (%)HIT, ±14 s (%)
EgoMemo12.312.37.77.7
GPM (ours)38.583.121.563.1

Under both tolerance settings, GPM clearly outperforms EgoMemo, a method designed specifically for EgoLife: under the tighter ±14 s window, GPM reaches 21.5% PERFECT and 63.1% HIT, versus 7.7% and 7.7% for EgoMemo. This shows that GPM's advantage holds on real memories: on naturally collected memories, GPM still retrieves the right memory and reminds at the right time.

Realism of the synthetic memories. We acknowledge that, to our knowledge, no publicly available egocentric dataset currently covers a user's life over a full year, which is necessary to build long-horizon personal memories; we therefore construct the personal histories synthetically. To make them as realistic as possible, realism is ensured during construction and then verified by human annotators.

  • Built from a coherent simulated life. To make the user's activities unfold realistically, memories are not isolated facts but are embedded in a full simulated life. We first synthesize a persona (biography, relationships, routines). Conditioned on it, we build a year-long event timeline with different categories of events: main-arc milestones, persona-derived interests, neutral daily distractors, and the evidence. Every event is then rendered into timestamped first-person captions ([V] visible action, [C] conversation). For example (App. D.3), the memory "Watering the basil straight into the middle caused the soil to crater; you decided to water around the edges next time" (~14 days before the trigger) is rendered into a realistic first-person caption stream of daily life (abbreviated):

    [V] 12:32:58 I stand in front of the potted basil plant on the windowsill. [V] 12:33:09 I see water pouring from the spout into the middle of the basil's soil. [V] 12:33:18 I see the soil begin to sink in, forming a small crater. [C] 12:33:28 I mutter to myself, "That crater's not great." [V] 12:33:44 I reach forward to touch the edge of the crater with my index finger. [C] 12:33:52 I mutter to myself, "Next time, around the edges."

    This episode is one of many everyday events in a ~618K-token year-long history, so, as in a real lifelog, the relevant moment must be found among many unrelated events.

  • Verified by human annotators. Annotators are shown both the abstract memory and the life stream into which it is rendered, and judge memory–event consistency: whether the rendered life stream is realistic and describes the same event as the memory. 98.4% of judgments answer yes.

AuthorsOpenReview ↗

Response to Reviewer wvxt (3/4)

Predefined trigger windows. The trigger window of every sample is ensured to be a moment worth intervening, first during construction and then by human review (App. C).

  • Filtered during construction. Each memory is generated for a specific target appearance and must pass a worth-triggering gate, which checks that the memory is grounded in the video, specific to the target, and that seeing the target genuinely warrants a reminder. Each sample then passes model-based quality control.

  • Verified by human annotators against explicit rubrics. 32 trained annotators watch the egocentric clip alongside the target, the memory, and the proposed trigger window; 87.7% of samples are reviewed by three independent annotators. They judge:

    1. Target visibility: is the target object needed for the assistance clearly visible and is it the described object? (95.8% yes)
    2. Trigger window: is the proposed window correct? Annotators keep it, adjust its boundaries, or mark it unusable. (82.7% kept, 14.2% adjusted, 3.1% unusable)
    3. Reason to remind: given the user's memory, is the reminder genuinely warranted at this moment? (93.0% yes)

    Each annotator then gives a verdict (accept, fix the window, or reject), and a sample enters the benchmark only if the majority accepts it (1,804 samples, 83.8% of those reviewed).

Summary. GPM's advantage holds on real memories: it clearly outperforms EgoMemo there under both tolerance settings, while the synthetic memories and trigger windows are built to be realistic and are verified by human annotators. Together, these support the realism of our benchmark and the effectiveness of GPM beyond synthetic data.

W3 & Q2. Consistency of the trigger-window definition and intermittent target appearances.

W3: "Inconsistent trigger-window definition: Section 3.1 and Appendix C appear to define trigger windows differently, raising concerns about the validity of timing-based evaluation."

Q2: "How are intermittent target appearances handled in the actual trigger-window annotations? Which rule is used by the released annotations and evaluation implementation? How many samples have multiple disjoint target appearances?"

Thank you for pointing this out, and we are sorry for the confusion. Both questions concern the same issue, so we answer them together. The two descriptions refer to the same window; we first clarify the full construction process of the trigger window, and then answer which rule is used and how many samples have disjoint appearances.

  • Locate the target appearance. We first locate the target object and scan it over the entire video, recording when it is visible. Brief gaps of at most 8 s (look-aways or occlusions) are merged, so one continuous appearance forms one window, from its first to its last visible timestamp (App. C, Step 1).
  • One appearance per clip. When cutting the test clips, we select each clip to contain exactly one such appearance. Hence, each clip has a single trigger window, and during the rest of the clip the system should remain silent.
  • Human verification. Annotators then check each window on the video and manually fine-tune the boundaries of windows they find inappropriate (14.2% adjusted).

Because each clip contains only one appearance, the "first appearance" in Sec. 3.1 and the "first and last visible timestamps" in App. C describe the same window. The first-appearance rule in Sec. 3.1 states the general principle, while App. C describes how this window is concretely built. The benefit of this design is that the right moment is unambiguous: the assistant should intervene at the earliest appropriate moment, and once it has passed, re-raising the same matter at a later reappearance would be a redundant interruption rather than a new opportunity to help. With a single window per clip, the ground truth is correct and unambiguous, and any fire outside it can be cleanly counted as early, late, or repeated triggering. Both the annotations and the evaluation use this single human-verified window. We will revise Sec. 3.1 and App. C to make this consistent.

Which rule is used, and how many samples have disjoint appearances? The released annotations and evaluation implementation use the single human-verified window of each clip. Since each clip is selected to contain exactly one target appearance (with gaps of at most 8 s merged), no sample has multiple disjoint target appearances. This single-window design guarantees that the ground truth is correct and unambiguous: each clip has exactly one moment where a reminder is warranted, so any fire outside this window can be cleanly counted as early, late, or repeated triggering. We additionally include 150 negative clips that contain no moment requiring a reminder at all, to evaluate whether the system stays silent.

Reviewer_XcGeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is motivated from the need of proactive assistant with memory rather than reactive assistant that can offer help at the right moment. The paper proposes memory-driven proactive assistance (MPA) which when given an egocentric video and a long-term memory, the system decides both when to speak and what to say. They build a benchmark to evaluate how current models and framework perform on the task and show that it needs the right memory at the right time to solve the problem. Based on this, the authors introduce Gated Proactive Memory, a plug-and-play harness that achieve strong performance on the benchmark.

Critical strengths
  • Proactive memory is an important direction and the proposed MPA benchmark provides a way to evaluate it
  • The paper proposes GPM, a plug and play harness for proactive assistance that performs well on the task.
Critical weaknesses
  • How is the trigger window or proactive timing chosen? Defining a trigger scenario seems to be ambiguous. For example, in Figure 1, some users may want the assistant to speak up while other would not, and it is unclear how the authors resolve such cases.
  • How is the width of the window decided? How much is the tolerance?
Questions for authors
  • Does the author have good intuition of why naive RAG with timing doesn't help so much?
  • How many distractors are in the video? How many right memory exists in the video?
  • How important is the two layers (episodic / semantic) in GPM? What is stored in each layer? Are the two necessary?
  • Is GPM improvement mainly due to being in-domain?
9 comments
AuthorsOpenReview ↗

Response to Reviewer XcGe (1/3)

Dear Reviewer XcGe,

We sincerely appreciate your valuable feedback and the thoughtful questions on our submission. We hope our rebuttal below addresses your concerns and merits a better score.

W1. How is the trigger window chosen? Ambiguity of whether to speak.

"How is the trigger window or proactive timing chosen? Defining a trigger scenario seems to be ambiguous. For example, in Figure 1, some users may want the assistant to speak up while other would not, and it is unclear how the authors resolve such cases."

Thank you for this great question. We agree that different users may want different levels of proactivity, and personalizing it is a promising future direction. To avoid such ambiguity in the benchmark, our construction makes sure that each trigger is clearly warranted by the user's memory, regardless of personal preference. The trigger is usually supported not by a single memory but by several memories together, which can be semantic and/or episodic. Taking the example in Figure 1, the user keeps sparkling water at home (semantic), and finished the last bottle yesterday, saying "I should restock sparkling water" (episodic). With these pieces of evidence combined, when passing sparkling water in a store, the user is very likely to expect a reminder, and we expect that this reminder is plausibly helpful to the user. We deliberately target such decisive, low-controversy triggers, and this is also verified by our annotators.

  • Guaranteed during construction. Each trigger scenario is built around a specific user: the memory events are generated for the target object needed for the assistance and must pass a worth-triggering gate, which checks that the memory is grounded in the video, specific to the target, and that seeing the target genuinely warrants a reminder for this user, given their memory. Whether to speak is therefore not left to a generic preference but is determined by the user's own history.
  • Verified by three annotators. Every sample is reviewed by human annotators, mostly by three independent annotators (87.7%), who watch the clip together with the memory, the reminder, and the proposed trigger window, and judge whether the reminder is warranted at this moment (93.0% yes). Annotators also check the trigger window itself and adjust it when it is not appropriate (14.2% adjusted, 3.1% marked unusable). A sample is kept only if the majority agrees, so every reminder in the final benchmark is one that annotators agree is right at this moment (App. C).

W2. Window width and tolerance.

"How is the width of the window decided? How much is the tolerance?"

We appreciate the reviewer's attention to this detail. We explain how the window width is determined and how much tolerance is used.

Window width. The window is the period during which the target object appears. We scan the target over the entire video and record when it is visible; brief gaps of at most 8 s (look-aways or occlusions) are merged, so one continuous appearance forms one window, from its first to its last visible timestamp. When cutting the test clips, we select each clip to contain exactly one such appearance, so each clip has a single trigger window, and during the rest of the clip the system should remain silent; this also avoids ambiguity about when to speak(App. C, Step 1). We additionally include 150 negative clips that contain no moment requiring a reminder at all. Annotators then check each window on the video and manually fine-tune the boundaries of windows they find inappropriate (14.2% adjusted).

Tolerance. The tolerance is 2 s after the end of the window, for two reasons. First, in our observation, within about 2 s after the trigger window ends, the user has usually already moved on to another object, so a reminder about the target is no longer timely. Second, the windowed (offline) paradigm makes a decision every 2 s, so a 2 s tolerance allows for one decision step. Moreover, our method rarely fires late (late-trigger 2.4% windowed and 3.0% streaming, vs. up to 62% for baselines), so the tolerance has little effect on our results.

AuthorsOpenReview ↗

Response to Reviewer XcGe (2/3)

Q1. Why naive RAG with timing doesn't help much.

"Does the author have good intuition of why naive RAG with timing doesn't help so much?"

This is an excellent question. We give our intuition from three aspects: why naive RAG still performs poorly even with timing given, why this motivates our structured graph-based memory, and how varying the retrieval depth kk affects naive RAG.

Why naive RAG with timing still performs poorly. With timing given, retrieval is triggered only inside the trigger window, and at each step it returns different relevant information. Most of it, however, is either retrieval noise or relevant but not worth-reminding information. Together, these lead the model to remind about the wrong thing, so even given the trigger window, naive RAG still performs poorly: in our controlled study (Sec. 4.2, Table 4), it reaches only 7.3% PERFECT on Qwen3.8-27B and 15.3% on GPT-5.3-Chat, versus 77.3% and 96.0% with the oracle memory under the same timing.

This is why we build a structured, graph-based memory. A structured memory naturally captures the links between an object and the events it is involved in, so seeing the object leads directly to its related events. Embedding similarity alone, in contrast, tends to match many memories that are semantically similar but actually irrelevant. Our ablation further confirms it: even with the gate kept, simply replacing our graph-based structured memory with RAG drops PERFECT from 37.3% to 20.7%.

Naive RAG with different kk does not help either. During the rebuttal, we additionally swept the retrieval depth kk of always-on naive RAG from 1 to 50 (Qwen3.8-27B, windowed). Here, no timing is given: the system retrieves the top-kk memories at every decision step and decides on its own whether to speak. We find that increasing kk does not improve performance. A small kk often misses the right memory, while a large kk brings in more distractors, so the system increasingly speaks when it should not, and PERFECT becomes even lower.

MethodkkPERFECT ↑Over-trigger ↓Fires/clip ↓
Naive RAG17.067.53.58
24.967.65.53
51.096.010.11
101.097.013.23
300.099.016.43
500.099.016.21
GPM (ours)–37.027.51.19

Our method achieves the best trade-off. GPM achieves the highest PERFECT and the lowest over-trigger rate while firing about once per clip, because its gate surfaces refined evidence only when it is worth intervening on (and none otherwise), instead of feeding top-kk raw passages at every step.

Q2. Number of distractors and right memories.

"How many distractors are in the video? How many right memory exists in the video?"

We appreciate the question; we clarify the two sides of each sample.

In the video. Each clip contains exactly one human-verified trigger window, i.e., one moment where a reminder is warranted; during the rest of the clip, the system should remain silent. The video itself is the user's current situation and does not contain memories.

In the memory. The "right memory" and the distractors live in the user's long-term history. Per sample, the history spans about a year and contains on average 946 events, 28.1K captions, and ~618K tokens. Only the target memory, the evidence that justifies the reminder, is the right memory: on average 1.2K tokens, i.e., only ~0.19% of the history (Table 2). Everything else, mostly neutral daily routines plus the user's other life threads, serves as distractors.

Memory history (per sample)Mean (min–max)
Time span (days)362 (330–365)
Events946 (641–1,020)
Captions28.1K (19.7K–34.3K)
Tokens618K (437K–762K)
Target (right) memory tokens1.2K (0.4K–3.4K)
Target memory share0.19% (0.06%–0.57%)

This also explains why our task is hard, and why conventional RAG methods struggle to perform well: the right memory is a tiny fraction buried in a long, distractor-rich history, and many distractors look semantically similar to the current scene.

AuthorsOpenReview ↗

Response to Reviewer XcGe (3/3)

Q3. Role of the episodic / semantic layers.

"How important is the two layers (episodic / semantic) in GPM? What is stored in each layer? Are the two necessary?"

Thank you for this insightful question. We first describe what each layer stores, and then show with a layer ablation that both are necessary.

What is stored in each layer. The episodic layer stores object-keyed memories of specific past events, i.e., what happened, so that an object in view directly retrieves related experiences. The semantic layer stores standing user facts such as habits, preferences, and constraints. In the sparkling-water example of Figure 1, "the user keeps sparkling water at home" is a semantic memory, while "finished the last bottle yesterday and said 'I should restock sparkling water'" is an episodic memory.

Are the two necessary? We ran a layer ablation on 200 samples (windowed, Qwen3.8-27B, harness without RL), changing only which memory layers the gate can see:

Memory layersPERFECT (%)HIT (%)
Full (episodic + semantic)39.553.0
Episodic only35.544.5
Semantic only5.05.0

We find that both layers play an important role. The episodic layer is essential: it captures the user's everyday actions and the links between objects and events, and without it PERFECT drops to 5.0%. The semantic memory is also helpful: adding it on top of the episodic layer raises PERFECT by 4.0 points (35.5% → 39.5%) and HIT by 8.5 points (44.5% → 53.0%), showing that the user's standing facts are important for completing the reminder task.

Q4. Is GPM's improvement due to being in-domain?

"Is GPM improvement mainly due to being in-domain?"

We appreciate this important question. We would like to clarify that our improvement comes from generalizable design rather than from being in-domain, which we support from three angles.

1. The harness is training-free and plug-and-play, and the benchmark is diverse. The GPM harness itself involves no training: it wraps an off-the-shelf model at inference time, and already lifts PERFECT on Qwen3.8-27B from 4.9% (naive RAG) to 37.0% (Table 5); wrapping GPT-5.3-Chat without any training likewise reaches 38%. Moreover, our benchmark covers many different scenes (kitchen & dining, retail store, workshop, living space, outdoors, utility room, office) and reminder reasons (task completion, verification, resource management, maintenance, object placement, safety), so the gains are not tied to one narrow domain.

2. Zero-shot transfer to an unseen dataset. We directly transfer GPM, without any adaptation, to 100 samples from the unseen Epic-Kitchens dataset. The harness lifts PERFECT from at most 11.0% (always-on RAG) to 41.0% windowed and 37.0% streaming, and the Ego4D-trained generator further reaches 49.0% and 41.0%, showing that the learned firing policy transfers across domains (Sec. 6.3, Figure 4a).

3. Transfer to real data and comparison with prior work. To further test transferability, we evaluate on EgoLife clips annotated by EgoServe, a different data source where both the egocentric video and the personal history (the user's own earlier recordings) are real. Unlike our benchmark, which annotates the full span from when the target object appears to when it disappears, EgoServe identifies a short trigger window and adds a ±60 s scoring tolerance on each side. Since most EgoServe services provide general assistance rather than memory-driven help (e.g., safety alerts, tool-use and next-step guidance), we keep only the memory-related types: Memory Recall, Memory Link, and Task Reminder. To ensure that the memory evidence lies outside both the test clip and the ±60 s tolerance window, we further keep only clips whose evidence occurs at least 3 minutes before the trigger, yielding 65 clips. The reminder can thus only be justified by memory rather than by the current view. We first follow EgoServe's ±60 s protocol. However, ±60 s is too loose for proactive reminders: a reminder up to a minute early or late still counts as correct, so being scored correct says little about whether it came at the right moment. We therefore additionally report a tighter ±14 s window, comparable to the median trigger-window length in our benchmark(24s+2s tolerance). We compare our GPM with EgoMemo, the graph-memory method proposed with EgoServe and designed for EgoLife:

MethodPERFECT, ±60 s (%)HIT, ±60 s (%)PERFECT, ±14 s (%)HIT, ±14 s (%)
EgoMemo12.312.37.77.7
GPM (ours)38.583.121.563.1

Even on real EgoLife data, GPM clearly outperforms EgoMemo, the method designed for this setting, under both tolerance settings, which validates the transferability of our method.

Reviewer_qFLiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory-driven proactive assistance, where an assistant needs to retrieve relevant information from long-term memory and decide when to intervene without an explicit user query. The authors introduce the MPA benchmark, combining egocentric videos with synthetic year-long personal memories, and propose Gated Proactive Memory (GPM) to improve memory retrieval and intervention timing.

Critical strengths

Proactive assistance is an important direction for real-world AI assistants, which should recognize when help is needed rather than simply responding to user questions.

The authors separate memory retrieval and intervention timing through controlled experiments, showing that both are important for successful proactive assistance. These findings provide useful insights for future research.

The benchmark considers not only the correctness of an intervention but also its timing and frequency. The experiments further include ablations and evaluations of unnecessary interventions.

Critical weaknesses

The authors generate year-long personal histories based on predefined reminder scenarios. While this provides controlled evaluation, it is unclear how well these synthetic histories reflect real user experiences and memory distributions. Could the authors further validate the realism of the generated memories and their relationship to the target interventions?

The benchmark evaluates whether an assistant responds within a predefined trigger window, but it is unclear whether it considers the user's ongoing activity. Even a relevant reminder could interrupt an important task. Could the authors clarify how the trigger windows account for the user's current activity and whether an intervention is actually appropriate?

Questions for authors

The authors should further clarify the differences between their benchmark and existing proactive assistance benchmarks, particularly EgoServe/EgoMemo, which also investigate proactive assistance using egocentric video and memory. A more direct comparison would help establish the novelty of the proposed setting and method.

The paper evaluates when an assistant should proactively intervene, but it is unclear how the system handles interruptions from the user. For example, if the user interrupts or dismisses a reminder while the assistant is speaking, can the assistant stop and continue its normal operation?

6 comments
AuthorsOpenReview ↗

Response to reviewer qFLi (1/3)

Dear Reviewer qFLi,

We sincerely thank you for your insightful review and constructive comments. We hope our responses below address your concerns and merit a better score.

W1. Realism of the synthetic memories and their relation to the target interventions.

"The authors generate year-long personal histories based on predefined reminder scenarios. While this provides controlled evaluation, it is unclear how well these synthetic histories reflect real user experiences and memory distributions. Could the authors further validate the realism of the generated memories and their relationship to the target interventions?"

We thank the reviewer for this insightful question. We address it from three aspects: (1) how the memories relate to the target interventions, (2) the realism of the synthetic memories, and (3) whether GPM's advantage holds on real memories.

Relation to the target interventions. The trigger window of every sample is ensured to be a moment worth intervening, first during construction and then by human review (App. C).

  • Filtered during construction. Each memory is generated for a specific target appearance and must pass a worth-triggering gate, which checks that the memory is grounded in the video, specific to the target, and that seeing the target genuinely warrants a reminder. Each sample then passes model-based quality control.

  • Verified by human annotators against explicit rubrics. 32 trained annotators watch the egocentric clip alongside the target, the memory, and the proposed trigger window; 87.7% of samples are reviewed by three independent annotators. They judge:

    1. Target visibility: is the target object needed for the assistance clearly visible, and is it the described object? (95.8% yes)
    2. Trigger window: is the proposed window correct? Annotators keep it, adjust its boundaries, or mark it unusable. (82.7% kept, 14.2% adjusted, 3.1% unusable)
    3. Reason to remind: given the user's memory, is the reminder genuinely warranted at this moment? (93.0% yes)

    Each annotator then gives a verdict (accept, fix the window, or reject), and a sample enters the benchmark only if the majority accepts it (1,804 samples, 83.8% of those reviewed).

Realism of the synthetic memories. We acknowledge that, to our knowledge, no publicly available egocentric dataset currently covers a user's life over a full year, which is necessary to build long-horizon personal memories; we therefore construct the personal histories synthetically. To make them as realistic as possible, realism is ensured during construction and then verified by human annotators.

  • Built from a coherent simulated life. To make the user's activities unfold realistically, memories are not isolated facts but are embedded in a full simulated life. We first synthesize a persona (biography, relationships, routines). Conditioned on it, we build a year-long event timeline with different categories of events: main-arc milestones, persona-derived interests, neutral daily distractors, and the evidence. Every event is then rendered into timestamped first-person captions ([V] visible action, [C] conversation). For example (App. D.3), the memory "Watering the basil straight into the middle caused the soil to crater; you decided to water around the edges next time" (~14 days before the trigger) is rendered into a realistic first-person caption stream of daily life (abbreviated):

    [V] 12:32:58 I stand in front of the potted basil plant on the windowsill. [V] 12:33:09 I see water pouring from the spout into the middle of the basil's soil. [V] 12:33:18 I see the soil begin to sink in, forming a small crater. [C] 12:33:28 I mutter to myself, "That crater's not great." [V] 12:33:44 I reach forward to touch the edge of the crater with my index finger. [C] 12:33:52 I mutter to myself, "Next time, around the edges."

    This episode is one of many everyday events in a ~618K-token year-long history, so, as in a real lifelog, the relevant moment must be found among many unrelated events.

  • Verified by human annotators. Annotators are shown both the abstract memory and the life stream into which it is rendered, and judge memory–event consistency: whether the rendered life stream is realistic and describes the same event as the memory. 98.4% of judgments answer yes.

GPM's advantage holds on real memories. (Next Page)

AuthorsOpenReview ↗

Response to reviewer qFLi (2/3)

GPM's advantage holds on real memories. To show that our method works on real memories, we further evaluate on EgoLife clips annotated by EgoServe, where both the egocentric video and the personal history (the user's own earlier recordings) are real. Unlike our benchmark, which annotates the full span from when the target object appears to when it disappears, EgoServe identifies a short trigger window and adds a ±60 s scoring tolerance on each side. Since most EgoServe services provide general assistance rather than memory-driven help (e.g., safety alerts, tool-use and next-step guidance), we keep only the memory-related types: Memory Recall, Memory Link, and Task Reminder. To ensure that the memory evidence lies outside both the test clip and the ±60 s tolerance window, we further keep only clips whose evidence occurs at least 3 minutes before the trigger, yielding 65 clips. The reminder can thus only be justified by memory rather than by the current view. We first follow EgoServe's ±60 s protocol. However, ±60 s is too loose for proactive reminders: a reminder up to a minute early or late still counts as correct, so being scored correct says little about whether it came at the right moment. We therefore additionally report a tighter ±14 s window, comparable to the median trigger-window length in our benchmark (24s+2s tolerance). We compare our GPM with EgoMemo, a graph-memory method designed for EgoLife:

MethodPERFECT, ±60 s (%)HIT, ±60 s (%)PERFECT, ±14 s (%)HIT, ±14 s (%)
EgoMemo12.312.37.77.7
GPM (ours)38.583.121.563.1

GPM clearly outperforms EgoMemo under both tolerance settings, showing that its advantage holds on real memories.

W2. Trigger windows and the user's ongoing activity.

"The benchmark evaluates whether an assistant responds within a predefined trigger window, but it is unclear whether it considers the user's ongoing activity. Even a relevant reminder could interrupt an important task. Could the authors clarify how the trigger windows account for the user's current activity and whether an intervention is actually appropriate?"

This is an important point, and we appreciate the reviewer raising it. We address it from three aspects: how the trigger window is tied to the user's ongoing activity by construction, how annotators verify that a reminder is appropriate at that moment, and how the LLM judge checks appropriateness at evaluation time.

  • By construction, the trigger window lies within the user's ongoing activity with the target. When building each sample, the target object is selected from what the user is currently doing in the clip: it is an object within the user's current activity (e.g., passing sparkling water on a shelf while shopping), and its appearance is a natural opportunity for a reminder (App. C, Step 1). The trigger window is therefore part of the user's ongoing activity, rather than an interruption of an unrelated task.

  • Annotators further check whether the current situation warrants a reminder. Annotators watch the egocentric clip while seeing the ground-truth reminder and the memory that justifies it, and judge the reminder against what the user is doing at that moment. They answer "Given the user's memory, is the reminder genuinely warranted at this moment?" (93.0% yes), and can adjust the window (14.2%) or mark it unusable (3.1%). Samples judged not appropriate are filtered out: only samples accepted by the majority of annotators enter the benchmark (1,804 of 2,153 reviewed, 83.8%; App. C).

  • The LLM judge also sees the current situation and the reminder together. At evaluation, the judge receives the emitted reminder, the current frame(s) around the firing time, and the memory, and outputs GOOD only if the reminder is correct for this moment and worth interrupting for. The judge prompt (App. H):

    System: You are an expert evaluator of a PROACTIVE assistant that watches the user's screen / first-person activity and, holding the user's long-term memory, speaks up to remind or help. Judge the QUALITY of ONE thing it said at a specific moment. [...] Evaluate THREE criteria:

    1. memory_grounded: does ANSWER correctly draw on the RIGHT memory, explicitly or implicitly, without distortion, rather than merely reacting to visible or recent context?
    2. correct: is it accurate and safe for THIS moment, with a valid memory-to-current-scene link and no hallucinated, distorted, harmful, or incorrect claim?
    3. useful: does it provide a concrete, actionable point whose specificity fits what is knowable now and is worth interrupting for?

    DECISION: if memory_grounded is false, output NON_MEMORY; otherwise, if correct is false, output BAD; otherwise, if useful is false, output BASIC; otherwise, output GOOD.

    User: ANSWER (judge this): {assistant reminder} / CURRENT FRAME(S): {image(s)} / MEMORY: {the correct stored memory}

AuthorsOpenReview ↗

Response to reviewer qFLi (3/3)

We further verify that this judge is reliable: we conducted human verification on 50 sampled clips: on the clip-level PERFECT outcome, the GPT-4o judge agrees with the human majority vote on 90.0% of clips (Cohen's κ = 0.79), showing that its judgments are consistent with humans.

Q1. Differences from EgoServe / EgoMemo, and a more direct comparison.

"The authors should further clarify the differences between their benchmark and existing proactive assistance benchmarks, particularly EgoServe/EgoMemo, which also investigate proactive assistance using egocentric video and memory. A more direct comparison would help establish the novelty of the proposed setting and method."

We appreciate this helpful suggestion. We acknowledge that EgoServe and EgoMemo are excellent, foundational concurrent work on proactive assistance with egocentric video and memory. Below, we clarify our differences and establish a direct comparison.

Setting differences. EgoServe studies short-horizon assistance within the same day; our MPA benchmark targets reminders that can only be justified by long-term personal memory, and evaluates them strictly.

EgoServe / EgoMemoMPA (ours)
MemoryBuilt by the system from the user's earlier video; memory-recall gaps ≤ 1 h (mostly)Year-long personal history (~618K tokens), distractor-rich
EvidenceMined by Gemini from EgoLife captionsExplicitly annotated and human-verified
ScoringA short identified trigger window plus a ±60 s tolerance on each side; content scored separatelyHuman-verified trigger window; content and timing scored jointly

A two-way comparison.

  • Our method on their benchmark. GPM vs. EgoMemo on EgoLife clips annotated by EgoServe (see W1):

    MethodPERFECT, ±60 s (%)HIT, ±60 s (%)PERFECT, ±14 s (%)HIT, ±14 s (%)
    EgoMemo12.312.37.77.7
    GPM (ours)38.583.121.563.1
  • Their method on our benchmark. We adapt EgoMemo to our MPA benchmark with its native pipeline and the paper's scoring rule, and evaluate on 100 randomly sampled clips. To match the generator backbone, we also run EgoMemo with GPM's decision model, Qwen3.8-27B:

    MethodPERFECT (%)HIT (%)
    EgoMemo (GPT-5.3 decision)4.08.0
    EgoMemo (Qwen3.8-27B decision)0.08.0
    GPM (ours, Qwen3.8-27B)38.055.0

GPM outperforms EgoMemo in both directions.

Why EgoMemo performs poorly on our data. EgoMemo's pipeline first looks at the current scene and then decides whether to retrieve memory. In our setting, however, the trigger moment is often not caused by the object itself: it is the user's memory that makes the object worth a reminder. Seeing the target alone rarely looks retrieval-worthy, so EgoMemo often chooses not to retrieve at the very objects that should trigger a reminder (it issues no reminder on 86% of clips), which leads to its low HIT and PERFECT. GPM instead keeps the relevant memory in view and then uses its gate to post-filter what to say based on that memory. This balances both sides: it does not miss memory-driven triggers, while the gate filters out unnecessary reminders.

Q2. Handling user interruptions or dismissals.

"The paper evaluates when an assistant should proactively intervene, but it is unclear how the system handles interruptions from the user. For example, if the user interrupts or dismisses a reminder while the assistant is speaking, can the assistant stop and continue its normal operation?"

Thank you for this practical question. We first clarify the scope of our paper, and then describe how GPM could be adapted to handle interruptions.

Our focus is the role of memory in deciding whether and when to intervene. How to combine reactive interaction (e.g., user interruptions, dismissals, or questions) with proactive behavior is not the focus of this paper, although we view it as an important direction for future work.

That said, our components can also potentially support such a system, and a potential adaptation is straightforward. If the user interrupts or dismisses a reminder, the assistant stops speaking and GPM marks the matter as closed in its record of what has already been said, so the gate will not raise it again; GPM then resumes normal monitoring. If the user asks a question, the assistant stops speaking and answers it using the same memory store and retrieval, then returns to proactive monitoring.

Comments on the paper

1 comment

Dear Reviewers,

The authors have responded to your comments. Could you please review their responses and update your reviews?

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.