acceptodds
Under review as a conference paper at ICLR 2027

WhenMemBench: Benchmarking Implicit Memory Need Recognition During Task Execution

Abstract

Long-term memory helps only when an agent recognizes that hidden history is needed, accesses it when its relevance becomes observable, and applies the right evidence. Existing memory benchmarks often reveal historical context, precompute retrieval results, or issue explicit recall requests; therefore, they evaluate memory use after the access decision has already been made. However, real users usually provide underspecified requests and may not know which prior detail matters. We formalize this gap as Implicit Memory Need Recognition: without an explicit memory recall request from the user, the visible state alone becomes insufficient for the agent to fulfill a task-critical requirement that implicitly calls for evidence from the hidden interaction prefix. We introduce WhenMemBench, an executable benchmark for this problem. To establish the benchmark, we convert multi-turn tool trajectories into auditable tasks through automated trajectory mining and naturalization followed by human-in-the-loop validation. The benchmark consists of 164 tasks, distinguishing Query-Triggered tasks, where historical relevance is inferable from the initial request, and Execution-Triggered tasks, where it emerges after workspace or tool observations. We further propose Step-Level Proactive Memory, a baseline strategy that uses a separate memory agent to selectively deliver relevant history at each execution step, and evaluate it among seven memory strategies on WhenMemBench using GPT-5.6. Our strategy achieves 43.29% overall task success, compared with 27.44% for an agent-controlled memory tool and 18.29% for Always-On RAG. These results reveal that implicit memory need recognition remains challenging for agents, even when the relevant history is retrievable from external memory.

Primary area:
datasets and benchmarks
Keywords:
memory need recognition, long-term memory, AI agents, benchmarks, proactive memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4BLhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WhenMemBench, a benchmark designed to evaluate implicit memory-need recognition during agent task execution. It consists of 164 executable tasks categorized into Query-Triggered and Execution-Triggered memory needs. The authors propose three evaluation metrics—Success Rate (SR), Memory Delivery Rate (MDR), and Timeliness Score (TS)--to assess task completion, memory delivery, and delivery timing. They also introduce Step-Level Proactive Memory, which uses a separate memory agent to retrieve and selectively inject relevant historical information. Experiments across seven memory strategies demonstrate improvements in task success, particularly for Execution-Triggered tasks.

Critical strengths
  1. Well-structured benchmark and evaluation. The benchmark provides executable tasks with traceable historical evidence and annotated memory-need boundaries. The proposed metrics offer useful insights into memory delivery and task performance.

  2. Interesting empirical findings. The experiments reveal substantial performance gaps between memory strategies, particularly on Execution-Triggered tasks, highlighting the potential benefits of proactive memory delivery.

Critical weaknesses
  1. Limited benchmark novelty. Existing benchmarks, such as MemoryArena, Mem2ActBench, and ClawArena, already evaluate implicit memory dependencies during agent execution without explicit recall instructions. The paper does not sufficiently differentiate its benchmark from these works or establish that its metrics directly measure memory-need recognition rather than memory delivery.

  2. Limited methodological novelty. The proposed Step-Level Proactive Memory closely resembles existing proactive memory frameworks, particularly Remember When It Matters, which already employs a separate memory agent for selective intervention. The methodological differences appear relatively incremental.

  3. Insufficient ablation studies. The ablations primarily examine initial-only monitoring and the removal of follow-up searches. They do not adequately isolate the effects of need recognition, retrieval refinement, and selective memory injection. More controlled comparisons with matched retrieval budgets and computational costs would strengthen the experimental evidence.

References

Questions for authors
  1. Benchmark novelty and validity: How does WhenMemBench fundamentally differ from MemoryArena, Mem2ActBench, and ClawArena in evaluating implicit memory needs? Since all 164 tasks require memory, how can the benchmark distinguish genuine need recognition from indiscriminate retrieval? Would memory-unnecessary tasks be needed to evaluate false-positive retrieval decisions?

  2. Methodological novelty: What are the key technical differences between Step-Level Proactive Memory and Remember When It Matters (Wu et al., 2026)? Can the authors clarify which components are novel and provide evidence that these components contribute independently to performance improvements?

Ablation and fairness: Can the authors provide more controlled ablations that independently examine proactive monitoring, retrieval refinement, and selective injection? How do the proposed method's gains change when compared with baselines under matched retrieval budgets, context-retention policies, and computational costs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4BLh (1/2): Benchmark and Evaluation

Thank you for asking us to clarify what the benchmark measures, how the reference baseline relates to prior work, and what each part of the baseline contributes.

W1 & Q1. What WhenMemBench measures

WhenMemBench pairs required historical evidence with the annotated point at which its relevance becomes observable, enabling evaluation of whether the right information arrives when the task needs it (Table 1 and Section 2). MemoryArena [1] evaluates experience across dependent sessions, Mem2ActBench [2] evaluates history-guided tool use, and ClawArena [3] evaluates adaptation to changing information. For example, an event-checklist task reveals in a workspace file that the venue is a temporary pop-up site; an earlier message defines the checklist for that venue type. The first tool observation revealing the venue type is the annotated need boundary.

MDR measures whether all required historical facts reach the Main Agent. TS measures how soon they are available together once the annotated need becomes observable, and SR checks whether the task requirements are met. These metrics evaluate the system's observable response to an annotated need: they distinguish missing or delayed evidence from task failure despite timely delivery.

To distinguish the task-level effects of selective and unfiltered injection, we evaluate SR alongside MDR and TS. Our All Candidates ablation retains the same retrieval rules but sends every retrieved candidate after each check. On 146 paired first-pass tasks under a 600-second limit, Full Proactive Memory achieved 43.2% SR versus 40.4% for All Candidates. All Candidates had slightly higher MDR (58.2% vs. 57.5%) and TS (47.7 vs. 44.6, on a 0–100 scale). Thus, broader evidence delivery did not yield higher task success in this comparison. The joint evaluation captures the task-level value of selective injection (details in Part 2, W3/Q3).

Yes, tasks that do not require history provide the complementary test of unnecessary intervention. Alongside the 164 memory-needed tasks, we ran a separate OpenClaw Terra control on 107 tasks solvable from the current request and workspace. Proactive Memory searched on all 102 of its scored tasks and sent at least one reminder on 97. On the 100 tasks scored under both policies, No Memory succeeded on 69 and Proactive Memory on 64. We count search and delivery separately because a private search can end without a reminder. The frequent reminders in this control expose unnecessary intervention that the memory-needed tasks alone cannot reveal.

References

[1] He et al. (2026). MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks.

[2] Shen et al. (2026). Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents.

[3] Ji et al. (2026). ClawArena: Benchmarking AI Agents in Evolving Information Environments.

AuthorsOpenReview ↗

Response to Reviewer 4BLh (2/2): Baseline and Ablations

W2 & Q2. The role and design of our reference baseline

Step-Level Proactive Memory is the reference baseline accompanying WhenMemBench. It provides a concrete policy for studying how agents obtain and use historical information during task execution, with the aim of informing future memory-system design.

The baseline makes three design choices explicit and testable: when to check for historical information, whether to refine retrieval, and which retrieved records to deliver. The ablations in Table 4 show higher SR with step-level checks and follow-up search than with their respective controls. The additional comparison in W3/Q3 finds that selective injection achieves higher SR with fewer Main-Agent input tokens than injecting all retrieved candidates. These comparisons connect specific design choices to task outcomes and resource use.

We build on the separate Memory Agent design of Wu et al. [1]. Their system updates a structured memory bank from the ongoing task. Our baseline retrieves from a fixed store of earlier interactions, using the current request and recent tool observations before each Main-Agent call. The Memory Agent can refine its search and select original records for delivery; delivered records remain available in later calls. This design allows us to trace retrieval and delivery relative to the annotated memory-need boundary and compare the effects of monitoring, retrieval refinement, and injection selection under a shared evaluation protocol.

W3 & Q3. Component contributions and controlled comparison

The full baseline achieves higher SR than each ablated variant on both Query- and Execution-Triggered tasks, supporting the contributions of continued monitoring, follow-up search, and injection filtering.

VariantQuery SR ↑Execution SR ↑Overall SR ↑
Full Proactive Memory41.23%48.00%43.29%
w/o Continued Monitoring33.33%36.00%34.15%
w/o Follow-up Search30.70%38.00%32.93%
w/o Injection Filtering39.47%38.00%39.02%

The table combines the existing ablations in Table 4 with the new injection-filtering ablation. It covers 114 Query-Triggered and 50 Execution-Triggered tasks, using one selected rollout per task, including recovery runs.

w/o Continued Monitoring retains the initial check and its follow-up searches but removes subsequent checks. w/o Follow-up Search retains per-step checks but permits one search per check. w/o Injection Filtering sends every unique record retrieved during a check, including when the selector would send nothing. These variants examine the checking schedule, retrieval refinement, and record selection, respectively.

Our added comparison uses w/o Injection Filtering as a controlled baseline. Both policies share the same retrieval limits—up to three searches per check and five results per search—and retain delivered records under the same context policy.

Across 146 paired first-pass tasks, Full Proactive Memory achieves 43.2% SR, compared with 40.4% without injection filtering. We report input-token usage for both agents to account for the memory process alongside the Main Agent. Across 145 pairs with usage records, mean Main-Agent input is 87.4k versus 131.5k tokens, and Memory-Agent input is 95.2k versus 97.4k, respectively. This comparison shows higher task success with selective injection under shared retrieval and retention rules, together with lower input-token usage.

References

[1] Wu et al. (2026). Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents.

Thank you for the detailed rebuttal and additional experiments, which clarify some of my concerns. However, my main concerns regarding benchmark validity and novelty remain unresolved, particularly whether the benchmark truly evaluates memory-need recognition. Moreover, the improvement from selective injection is relatively small. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

What WhenMemBench Measures

Thank you for the follow-up. We agree that our metrics do not establish whether an agent internally recognized a need for memory. We use “memory-need recognition” in a behavioral sense: once the current task provides evidence that a past instruction applies, does the relevant memory reach the agent in time to use it?

An agent may retrieve the venue rule before opening the file, for instance, but only the file reveals whether it applies to this task. Our benchmark contribution is to annotate that point in an executable task, so eventual retrieval, timing, and task success can be evaluated separately.

In our experiments, selective injection yielded modest gains. The benchmark lets us examine those gains by tracking when relevant memories reach the agent and whether it completes the task.

Reviewer_2t2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies implicit memory need recognition: whether a tool-using agent recognizes, without an explicit recall request, that a task requires information from hidden past interactions, and brings that information into its context once the need becomes observable. It introduces WhenMemBench, a benchmark of executable tasks built from ClawArena histories. The tasks are divided into query-triggered tasks, where the need is inferable from the request, and execution-triggered tasks, where it emerges from tool observations during execution. The evaluation reports task success, memory delivery rate, and timeliness score. The paper also proposes Step-Level Proactive Memory, in which a separate memory agent retrieves, filters, and injects historical records before every step of the main agent. The authors report that this method achieves higher task success than other baselines and an agent-controlled memory tool.

Critical strengths
  1. The paper highlights a meaningful aspect of memory use in agents. Which is that the need for past information can emerge during execution rather than being apparent from the initial request. Distinguishing when a memory need becomes observable, and measuring whether and when the required evidence actually reaches the agent, offers a useful perspective that goes beyond evaluating retrieval quality or final task success alone.
Critical weaknesses
  1. The novelty of the work depends on the execution-triggered setting, which is not sufficiently justified. Recognizing and applying past information without an explicit recall request has already been studied in long-term memory research. For example, PersonaMem [1] evaluates whether models use information from earlier sessions to respond to new user queries that do not refer to it explicitly, and Mem2ActBench evaluates whether tool-using agents proactively apply constraints from earlier interactions to underspecified requests. Deciding when to retrieve as needs arise during a process has been studied in retrieval-augmented generation, for example by FLARE [2] and DRAGIN [3]. In addition, the proposed method adopts the separate memory-agent design of prior work, as the authors acknowledge. The distinctive contribution therefore lies in needs that become observable only during execution. For this problem definition to hold, it should first be shown that such needs cannot be anticipated from the request alone; if they can, the setting reduces to the existing implicit-memory problem at query time. However, in the paper’s own results, retrieving once from the initial request already delivers the complete evidence for roughly half of the execution-triggered tasks. It should also be shown that checking for memory needs throughout execution outperforms strong approaches that retrieve selectively once before execution, such as LLM-based query rewriting or memory selection, or existing implicit-memory strategies, yet such approaches are not included among the compared baselines. Moreover, the advantage of the proposed method over the agent-controlled memory tool comes mainly from the execution-triggered subset, which is small and partly synthetic. On query-triggered tasks, the memory tool matches or exceeds the proposed method for some of the additional models, and these results are not discussed.

  2. It is unclear whether every annotated memory need is a genuine requirement, which makes the binary evaluation difficult to interpret. The hidden history includes not only hard constraints but also softer information, such as conventions, priorities, and preferences. Whether such information must be proactively retrieved for a given request is often ambiguous, since the request could reasonably be satisfied with or without it. Nevertheless, it is treated as required evidence, and MDR score each rollout as a binary outcome depending on whether this information is delivered. Although the rubric marks some preferences as non-blocking, the paper does not explain how this distinction is made, whether such preferences still count as required evidence for MDR, or how many tasks involve them.

[1] Jiang, Bowen, et al. "Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale." arXiv preprint arXiv:2504.14225 (2025).

[2] Jiang, Zhengbao, et al. "Active retrieval augmented generation." Proceedings of the 2023 conference on empirical methods in natural language processing. 2023.

[3] Su, Weihang, et al. "Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Questions for authors
  1. How does WhenMemBench differ from prior work on using past information without explicit recall requests and from adaptive retrieval methods beyond the execution-triggered setting? Could the authors also report a performance comparison between the proposed method and strong approaches that retrieve selectively once before execution, such as LLM-based query rewriting, LLM selection over a larger candidate pool, or existing implicit-memory strategies?

  2. How do the authors decide whether historical information such as a convention, priority, or preference is a genuine requirement that must be proactively retrieved for a given request? How are blocking and non-blocking information distinguished, and is non-blocking information included in the required evidence for MDR? Could the authors provide concrete examples of borderline cases and how they were resolved?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2t2D (1/2): Benchmark and Initial Retrieval

Thank you for highlighting the value of evaluating when memory needs become observable. We address the benchmark’s contribution, initial retrieval, Query results, and requirement annotations below.

W1 & Q1: Contribution beyond implicit memory use

WhenMemBench connects long-term memory use to observable task needs. Each task links historical evidence, the point at which its relevance becomes observable, and the requirements of an executable task. MDR evaluates whether the evidence reaches the agent, TS evaluates its arrival time relative to that point, and SR evaluates the resulting artifact.

Revised Section 2 and Table 1 explain this evaluation coverage alongside existing memory benchmarks. Adaptive retrieval methods such as FLARE and DRAGIN develop policies for retrieving information during generation; WhenMemBench provides task-grounded annotations for evaluating memory delivery and its timing. This distinction also matters for Query tasks. For Terra, Memory Tool and Proactive Memory have similar MDR, 61.40% and 60.53%, but different TS, 17.63 and 50.27 on a 0–100 scale. The timing metric distinguishes behavior that eventual delivery rates alone do not capture.

W1 & Q1: Execution-triggered needs and initial retrieval

Consider the released activity-checklist case. The current request asks the agent to complete an activity’s submission checklist from a file. A prior conversation, stored in long-term memory, specifies that temporary pop-up venues require an attendance sheet, dated photographs, and an organizer’s statement instead of a permanent-venue registration certificate. Reading the current file reveals that this activity uses a temporary pop-up venue. The agent must connect that observation with the historical instruction to complete the checklist correctly.

An initial search can retrieve this rule. The venue type revealed by the file determines whether the rule applies to the current activity.

Our Initial-only control uses the same Memory Agent for one check before the first Main-Agent call, with follow-up searches and candidate selection available within that check. On Execution tasks, Table 4 reports SR of 36% for Initial-only versus 48% for full Proactive Memory, and MDR of 50% versus 78%. This comparison supports continued checking under the evaluated configuration. The benchmark includes 50 Execution tasks among 164 tasks, drawing on ClawArena histories and workspaces alongside synthetic scenarios.

W1: Query results across models

Table 3 shows that Memory Tool achieves higher Query SR than Proactive Memory with Opus and Qwen. Query requests already provide cues for accessing relevant history, which the Main Agent can use directly.

A supplementary Qwen trace comparison illustrates a concrete difference. The task asks the agent to prepare an HR tracker of an employee’s outstanding offboarding items. In an earlier conversation, the employee reported that the laptop and badge had not been returned and the final month’s salary remained unpaid. Memory Tool retrieved this message and included the outstanding items. Proactive Memory missed the message and omitted the unpaid-salary item. This case identifies a difference in retrieved evidence and final task coverage. Together with the aggregate results, it supports reporting Query and Execution performance separately.

AuthorsOpenReview ↗

Response to Reviewer 2t2D (2/2): Requirement Annotation and MDR

W2 & Q2: Required historical information and non-blocking details

Appendix B.3 and Table A2 describe our completed two-stage human review. Two reviewers independently assessed each task. They first judged whether and when the historical evidence was needed, without seeing the construction label, reference answer, or model results. If the task could be completed from the query and current materials without that evidence, they classified it as requiring no memory. They also flagged queries that disclosed the historical information. They then checked the reference answer against the task and evidence, identifying overly strict requirements or missing dependencies. Disagreements were resolved through third-reviewer adjudication.

A blocking requirement specifies a decision or content necessary for task success. Non-blocking annotations specify which details may be omitted from the final deliverable (Appendix B.2). For MDR, the required information consists of the decision-relevant facts and constraints in each task's annotated historical evidence (Section 3.3). MDR checks whether this information reached the Main Agent and accepts faithful paraphrases, summaries, and translations.

In the staffing task, the agent completes a personnel-placement file using its suggested values as defaults. A prior conversation records a manager’s proposal: if a particular candidate joins, they should enter a specified team as a P7 technical lead. When the file concerns this candidate, the agent needs that historical proposal to fill in the placement correctly. The team, role, and conditional status must be preserved; equivalent wording is accepted.

In the trading-monitoring task, the agent fills in launch decisions and follow-up actions for a spreadsheet containing a strategy with 97% monitoring coverage. A prior conversation states that every transaction must be monitored and mentions Grafana and Prometheus as the tools in use. Combining the current coverage figure with the historical requirement leads to postponing launch until monitoring is complete. The output must reflect that decision, but need not repeat the tool names.

Reviewer_A5KWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WhenMemBench, a benchmark for implicit memory need recognition that distinguishes two types of tasks: (1) Query-Triggered: The current request already identifies the relevant project or entity, making the potential relevance of historical information apparent at the beginning of the task. (2) Execution-Triggered: The agent must first inspect files or tool outputs to identify the relevant project or entity before the relevance of historical information becomes apparent.

The benchmark contains 164 tasks, including 114 query-triggered tasks and 50 execution-triggered tasks. It is built primarily from ClawArena histories and workspaces, supplemented with synthetic scenarios. Each task includes annotated historical evidence, task requirements, an observable boundary at which the memory need arises, and an outcome checker.

Critical strengths
  • The distinction between trigger types has diagnostic value. Separating query-triggered from execution-triggered memory needs better reflects the challenges faced by agents working across tasks and sessions than simply testing whether they can answer questions about past interactions.

  • The evaluation focuses on actual execution. Evidence found by a retriever does not count as successfully delivered unless it reaches the Main Agent's context. This helps distinguish bottlenecks in retrieval, delivery, and application.

  • The construction and review procedures are described in detail. They include historical provenance checks, checks that hidden requirements are not exposed in the visible task context, independent review by two reviewers, and adjudication by a third reviewer.

Critical weaknesses
  • The interaction between the evaluated memory policies and OpenClaw's native memory mechanisms is insufficiently specified.

  • Generalization across agent harnesses remains unclear. Although the paper evaluates four underlying models, all experiments use OpenClaw. It is therefore difficult to determine how much the observed differences between memory policies depend on this harness's prompts, tool configuration, execution loop, and context management. It remains unclear whether the key findings would hold in other harnesses, such as Claude Code and Codex.

  • The analysis provides limited guidance for designing future memory systems. The paper presents extensive empirical results across models, memory policies, and evaluation metrics, but the analysis primarily compares their performance without sufficiently explaining how these observations should inform system design. In particular, the performance gains of Proactive Memory need a more detailed mechanistic explanation. Although the paper includes ablations of step-level monitoring and follow-up search, it does not sufficiently characterize when and why the method is effective across different models and task types.

Questions for authors
  • How do the evaluated memory policies interact with OpenClaw's native memory mechanisms? Which native mechanisms are enabled, disabled, or replaced in each experimental condition?

  • Do the key findings hold in other agent harnesses, such as Claude Code and Codex? Could you provide evidence or analysis that distinguishes the effects of the memory policies from those of OpenClaw's prompts, tool configuration, execution loop, and context management?

  • Beyond the existing ablations, what mechanisms account for the performance gains of Proactive Memory, and what concrete lessons do these findings provide for the design of future memory systems?

  • Under which model and task conditions is Proactive Memory most effective, and when do its advantages diminish? Could you provide a more detailed analysis of these differences across models and trigger types?

2 comments
AuthorsOpenReview ↗

Response to Reviewer A5KW

Thank you for highlighting the value of separating the two trigger types and checking what evidence actually reaches the Main Agent.

W1 & Q1. OpenClaw memory configuration

We have added the OpenClaw memory configuration to Appendix F. In the GPT-5.6 Terra runs, all seven conditions retain within-session history and safeguard compaction. The plugin allowlist excludes the native memory-core plugin. Memory Tool, Proactive Memory, and Gold Injection use our experiment plugin, with memory_search exposed to the Main Agent only in Memory Tool.

The policies supply historical evidence through the access rules in Table 2 of the revised manuscript. No Memory supplies none. Mem0 and Query-only RAG retrieve once at the start and retain the results. Always-On RAG refreshes retrieval before each Main-Agent call. Memory Tool lets the Main Agent decide when to search. Proactive Memory uses a separate Memory Agent to select evidence before each Main-Agent call. Gold Injection supplies the annotated evidence in the initial input.

W2 & Q2. Results in Codex

We have added a Codex comparison to Section 5.2, with settings and full results in Appendix F.1 (Table A4). We evaluated Memory Tool and Proactive Memory on all 164 tasks in Codex CLI 0.164.0. Both conditions use GPT-5.6 Terra as the Main Agent at xhigh reasoning effort, the same starting workspaces and historical records, and the same retriever with five results per search.

In Memory Tool, the Main Agent decided when to call memory_search, and the results entered Codex’s normal tool history. In Proactive Memory, a separate Terra Memory Agent checked the task and up to four recent tool-action/result pairs before each Main-Agent call. It could search up to three times per check. The selected records were added as reminders and kept in subsequent inputs.

Codex methodQuery-Triggered (114): SR / MDR / TSExecution-Triggered (50): SR / MDR / TSOverall (164): SR / MDR / TS
Memory Tool42.1% / 44.7% / 7.44.0% / 6.0% / 1.530.5% / 32.9% / 5.6
Proactive Memory43.9% / 56.1% / 41.742.0% / 72.0% / 67.143.3% / 61.0% / 49.5

TS is shown on a 0–100 scale. Each task contributes one selected rollout per policy, shared by all three metrics.

The 328 Codex rollouts are scored using GPT-5.6 Sol. SR combines an artifact judgment with blocking deterministic checks. MDR and TS share one fact-level assessment per trace of the required gold facts supported by Main-visible memory. MDR checks cumulative delivery of all required facts. TS measures the delay from the need boundary until all required facts are jointly visible, with zero delay when they are already available at the boundary.

On Execution-Triggered tasks, Proactive Memory raises success from 2/50 to 21/50. On Query-Triggered tasks, success moves from 48/114 to 50/114. With the base model, harness, and task inputs shared within Codex, the larger gain on Execution-Triggered tasks also appears in a second harness for Terra.

The anonymous repository provides all benchmark tasks, per-task results, evaluation scripts, and a paired Memory Tool/Proactive Memory case.

W3 & Q3–Q4. Why and when Proactive Memory helps

Q3. How the gain arises. In a Codex task, the agent must complete an activity submission checklist. The workspace identifies a temporary pop-up site; prior history specifies three required items: a sign-in sheet, dated photos, and an organizer’s statement. Memory Tool reads the workspace but makes no memory lookup. Its final checklist omits the latter two items. Proactive Memory retrieves the rule before the first Main-Agent call and retains it when the site type becomes visible, so the final checklist includes all three. SR / MDR / TS are 0 / 0 / 0 for Memory Tool and 1 / 1 / 1 for Proactive Memory.

This case suggests two practical design choices: allow a separate memory process to initiate retrieval when the Main Agent does not search, and retain the selected evidence across later execution steps. paired traces and results.

Q4. When the gain is larger. In the OpenClaw results (Table 3 of the revised manuscript), Proactive Memory improves Execution-Triggered SR over Memory Tool by 54 percentage points for Opus, 46 for Terra, 22 for DeepSeek, and 10 for Qwen. The SR gains on Query-Triggered tasks are smaller for each model. On that subset, Proactive Memory improves TS for all four models, while Memory Tool has higher SR for Opus and Qwen. The initial request already provides a retrieval cue on Query-Triggered tasks. The strongest gains therefore occur when the need for history emerges during execution. For Terra in Codex, the TS gain is 65.6 points on Execution-Triggered tasks and 34.3 points on Query-Triggered tasks.

Thanks for the detailed rebuttal and additional experiments. I have carefully read the responses and considered the other reviewers’ comments. While the clarifications address some of my concerns, I remain unconvinced that the paper offers sufficient novelty and a substantial enough contribution beyond existing work on implicit memory use and proactive memory mechanisms. I therefore maintain my original score as weak reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.