- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WhenMemBench, a benchmark designed to evaluate implicit memory-need recognition during agent task execution. It consists of 164 executable tasks categorized into Query-Triggered and Execution-Triggered memory needs. The authors propose three evaluation metrics—Success Rate (SR), Memory Delivery Rate (MDR), and Timeliness Score (TS)--to assess task completion, memory delivery, and delivery timing. They also introduce Step-Level Proactive Memory, which uses a separate memory agent to retrieve and selectively inject relevant historical information. Experiments across seven memory strategies demonstrate improvements in task success, particularly for Execution-Triggered tasks.
Critical strengths
-
Well-structured benchmark and evaluation. The benchmark provides executable tasks with traceable historical evidence and annotated memory-need boundaries. The proposed metrics offer useful insights into memory delivery and task performance.
-
Interesting empirical findings. The experiments reveal substantial performance gaps between memory strategies, particularly on Execution-Triggered tasks, highlighting the potential benefits of proactive memory delivery.
Critical weaknesses
-
Limited benchmark novelty. Existing benchmarks, such as MemoryArena, Mem2ActBench, and ClawArena, already evaluate implicit memory dependencies during agent execution without explicit recall instructions. The paper does not sufficiently differentiate its benchmark from these works or establish that its metrics directly measure memory-need recognition rather than memory delivery.
-
Limited methodological novelty. The proposed Step-Level Proactive Memory closely resembles existing proactive memory frameworks, particularly Remember When It Matters, which already employs a separate memory agent for selective intervention. The methodological differences appear relatively incremental.
-
Insufficient ablation studies. The ablations primarily examine initial-only monitoring and the removal of follow-up searches. They do not adequately isolate the effects of need recognition, retrieval refinement, and selective memory injection. More controlled comparisons with matched retrieval budgets and computational costs would strengthen the experimental evidence.
References
- He et al. (2026). MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. ICML 2026. https://arxiv.org/abs/2602.16313
- Shen et al. (2026). Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents. https://arxiv.org/abs/2601.19935
- Ji et al. (2026). ClawArena: Benchmarking AI Agents in Evolving Information Environments. https://arxiv.org/abs/2604.04202
- Wu et al. (2026). Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents. https://arxiv.org/abs/2607.08716
Questions for authors
-
Benchmark novelty and validity: How does WhenMemBench fundamentally differ from MemoryArena, Mem2ActBench, and ClawArena in evaluating implicit memory needs? Since all 164 tasks require memory, how can the benchmark distinguish genuine need recognition from indiscriminate retrieval? Would memory-unnecessary tasks be needed to evaluate false-positive retrieval decisions?
-
Methodological novelty: What are the key technical differences between Step-Level Proactive Memory and Remember When It Matters (Wu et al., 2026)? Can the authors clarify which components are novel and provide evidence that these components contribute independently to performance improvements?
Ablation and fairness: Can the authors provide more controlled ablations that independently examine proactive monitoring, retrieval refinement, and selective injection? How do the proposed method's gains change when compared with baselines under matched retrieval budgets, context-retention policies, and computational costs?
4 comments
Response to Reviewer 4BLh (2/2): Baseline and Ablations
W2 & Q2. The role and design of our reference baseline
Step-Level Proactive Memory is the reference baseline accompanying WhenMemBench. It provides a concrete policy for studying how agents obtain and use historical information during task execution, with the aim of informing future memory-system design.
The baseline makes three design choices explicit and testable: when to check for historical information, whether to refine retrieval, and which retrieved records to deliver. The ablations in Table 4 show higher SR with step-level checks and follow-up search than with their respective controls. The additional comparison in W3/Q3 finds that selective injection achieves higher SR with fewer Main-Agent input tokens than injecting all retrieved candidates. These comparisons connect specific design choices to task outcomes and resource use.
We build on the separate Memory Agent design of Wu et al. [1]. Their system updates a structured memory bank from the ongoing task. Our baseline retrieves from a fixed store of earlier interactions, using the current request and recent tool observations before each Main-Agent call. The Memory Agent can refine its search and select original records for delivery; delivered records remain available in later calls. This design allows us to trace retrieval and delivery relative to the annotated memory-need boundary and compare the effects of monitoring, retrieval refinement, and injection selection under a shared evaluation protocol.
W3 & Q3. Component contributions and controlled comparison
The full baseline achieves higher SR than each ablated variant on both Query- and Execution-Triggered tasks, supporting the contributions of continued monitoring, follow-up search, and injection filtering.
| Variant | Query SR ↑ | Execution SR ↑ | Overall SR ↑ |
|---|---|---|---|
| Full Proactive Memory | 41.23% | 48.00% | 43.29% |
| w/o Continued Monitoring | 33.33% | 36.00% | 34.15% |
| w/o Follow-up Search | 30.70% | 38.00% | 32.93% |
| w/o Injection Filtering | 39.47% | 38.00% | 39.02% |
The table combines the existing ablations in Table 4 with the new injection-filtering ablation. It covers 114 Query-Triggered and 50 Execution-Triggered tasks, using one selected rollout per task, including recovery runs.
w/o Continued Monitoring retains the initial check and its follow-up searches but removes subsequent checks. w/o Follow-up Search retains per-step checks but permits one search per check. w/o Injection Filtering sends every unique record retrieved during a check, including when the selector would send nothing. These variants examine the checking schedule, retrieval refinement, and record selection, respectively.
Our added comparison uses w/o Injection Filtering as a controlled baseline. Both policies share the same retrieval limits—up to three searches per check and five results per search—and retain delivered records under the same context policy.
Across 146 paired first-pass tasks, Full Proactive Memory achieves 43.2% SR, compared with 40.4% without injection filtering. We report input-token usage for both agents to account for the memory process alongside the Main Agent. Across 145 pairs with usage records, mean Main-Agent input is 87.4k versus 131.5k tokens, and Memory-Agent input is 95.2k versus 97.4k, respectively. This comparison shows higher task success with selective injection under shared retrieval and retention rules, together with lower input-token usage.
References
[1] Wu et al. (2026). Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents.
Thank you for the detailed rebuttal and additional experiments, which clarify some of my concerns. However, my main concerns regarding benchmark validity and novelty remain unresolved, particularly whether the benchmark truly evaluates memory-need recognition. Moreover, the improvement from selective injection is relatively small. Therefore, I will maintain my original rating.
What WhenMemBench Measures
Thank you for the follow-up. We agree that our metrics do not establish whether an agent internally recognized a need for memory. We use “memory-need recognition” in a behavioral sense: once the current task provides evidence that a past instruction applies, does the relevant memory reach the agent in time to use it?
An agent may retrieve the venue rule before opening the file, for instance, but only the file reveals whether it applies to this task. Our benchmark contribution is to annotate that point in an executable task, so eventual retrieval, timing, and task success can be evaluated separately.
In our experiments, selective injection yielded modest gains. The benchmark lets us examine those gains by tracking when relevant memories reach the agent and whether it completes the task.
Response to Reviewer 4BLh (1/2): Benchmark and Evaluation
Thank you for asking us to clarify what the benchmark measures, how the reference baseline relates to prior work, and what each part of the baseline contributes.
W1 & Q1. What WhenMemBench measures
WhenMemBench pairs required historical evidence with the annotated point at which its relevance becomes observable, enabling evaluation of whether the right information arrives when the task needs it (Table 1 and Section 2). MemoryArena [1] evaluates experience across dependent sessions, Mem2ActBench [2] evaluates history-guided tool use, and ClawArena [3] evaluates adaptation to changing information. For example, an event-checklist task reveals in a workspace file that the venue is a temporary pop-up site; an earlier message defines the checklist for that venue type. The first tool observation revealing the venue type is the annotated need boundary.
MDR measures whether all required historical facts reach the Main Agent. TS measures how soon they are available together once the annotated need becomes observable, and SR checks whether the task requirements are met. These metrics evaluate the system's observable response to an annotated need: they distinguish missing or delayed evidence from task failure despite timely delivery.
To distinguish the task-level effects of selective and unfiltered injection, we evaluate SR alongside MDR and TS. Our All Candidates ablation retains the same retrieval rules but sends every retrieved candidate after each check. On 146 paired first-pass tasks under a 600-second limit, Full Proactive Memory achieved 43.2% SR versus 40.4% for All Candidates. All Candidates had slightly higher MDR (58.2% vs. 57.5%) and TS (47.7 vs. 44.6, on a 0–100 scale). Thus, broader evidence delivery did not yield higher task success in this comparison. The joint evaluation captures the task-level value of selective injection (details in Part 2, W3/Q3).
Yes, tasks that do not require history provide the complementary test of unnecessary intervention. Alongside the 164 memory-needed tasks, we ran a separate OpenClaw Terra control on 107 tasks solvable from the current request and workspace. Proactive Memory searched on all 102 of its scored tasks and sent at least one reminder on 97. On the 100 tasks scored under both policies, No Memory succeeded on 69 and Proactive Memory on 64. We count search and delivery separately because a private search can end without a reminder. The frequent reminders in this control expose unnecessary intervention that the memory-needed tasks alone cannot reveal.
References
[1] He et al. (2026). MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks.
[2] Shen et al. (2026). Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents.
[3] Ji et al. (2026). ClawArena: Benchmarking AI Agents in Evolving Information Environments.