- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LongMemEval-V2, a benchmark designed to evaluate whether agents can accumulate and use reusable environment-specific experience from large collections of historical web interaction trajectories. It contains 451 questions covering five abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. The questions are evaluated against histories of up to approximately 500 trajectories, or 115M tokens. Under the Context Gathering protocol, a memory system first processes the historical trajectories and then retrieves evidence for a given query, which a fixed reader uses to generate the final answer. The paper also introduces two baselines, AgentRunbook-R and AgentRunbook-C. AgentRunbook-C achieves accuracies of 74.9% on Small and 70.1% on Medium, offering a reasonable trade-off between accuracy and query latency.
Critical strengths
- Meaningful research problem. Long-term agent memory should capture not only user preferences but also reusable, environment-specific operational experience. The paper clearly identifies and focuses on this problem.
- Large-scale benchmark. LME-V2 includes 25.6M–114.8M tokens of historical data and links questions to multiple web-agent trajectories. Compared with benchmarks based on conversational histories, this setup better reflects the needs of long-running web and enterprise agents.
- Clear presentation. The paper is well written and easy to follow.
Critical weaknesses
- Limited benchmark novelty. The distinction from existing agent-memory benchmarks is not sufficiently clear. For example, AMA-Bench already evaluates long-term memory over agent trajectories and includes a memory system based on causal graphs and tool-augmented retrieval. The paper should make a clearer comparison and explain which unique capabilities LME-V2 measures and why existing benchmarks cannot adequately evaluate them.
- Limited domain coverage. The benchmark is restricted to customized web-browsing environments and a small set of websites in WebArena and WorkArena. It does not cover other agent settings, such as operating-system agents, embodied agents, or coding agents.
- Limited diversity in memory approaches. Both AgentRunbook baselines rely on retrieval and tool use. The paper does not explore trainable memory, memory distillation, or continual memory updates. It also lacks direct experimental comparisons with established agent-memory systems such as MemGPT, Mem0, and Memory-R1; these systems are only briefly discussed in the related work.
- Some results need clarification. The introduction reports 40.1% accuracy for the basic query-to-slice RAG baseline, whereas Table 2 reports 42.8% on Small and 38.1% on Medium, giving an unweighted average of 40.45%. The paper should clarify how the reported figure is calculated. In addition, the mapping between the five memory abilities and the category-level results in Table 2 is unclear, particularly for premise-awareness questions.
Questions for authors
See the weaknesses above.
2 comments
Thank you for the rebuttal. The response partially addresses my concerns. I will keep my score unchanged for now and discuss the dataset’s novelty, practical value, and rigor with the other reviewers in the next phase.
Thank you for the review
Thank you for recognizing the importance of the problem and our effort to build a large benchmark. We address your concerns below.
AMA-Bench [5] is concurrent work that focuses on memory within a task episode. It tests recall, causal inference, state updates, and state abstraction. LME-V2 asks what agents learn from 100-500 past trajectories, with explicit tests of reusable workflows, environment gotchas, and wrong premises. This adds the challenge of finding and combining relevant experience among many unrelated tasks. We will make these differences clearer.
We agree that broader domains would be valuable. We chose customized websites because models cannot reliably answer questions about their specific details from prior knowledge alone (Figure 4). We also explored coding tasks, but found that models often already memorized the repositories, making it harder to construct questions that require memory of past experience. Extending the benchmark to coding, OS, and embodied agents is an important direction.
Our main contribution is the benchmark. RAG pipelines and coding-agent memory provide two practical starting points, and AgentRunbook shows how each can be improved through memory design. The table summarizes differences in what existing systems preserve. For example, keeping only strategy notes can lose details needed for questions about specific UI fields.
Trainable memory, distillation, and continual updates are important directions for this benchmark. Memory-R1 [6], for example, learns memory operations through reinforcement learning. Our Insert/Query interface allows such approaches, and we hope LME-V2 will encourage their evaluation.
Thank you for spotting the inconsistency. The displayed Table 2 values average to 40.45%: (42.8% + 38.1%) / 2. We will correct the inconsistent number in the introduction and explain the aggregation.
As shown in Figure 2, premise-awareness questions are wrong-premise variants of static, dynamic, and workflow questions. Table 2 includes each variant in its corresponding category:
We will make this mapping explicit in the Table 2 caption.
References
[1] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.
[2] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.
[3] Xu et al. A-MEM: Agentic Memory for LLM Agents.
[4] Packer et al. MemGPT: Towards LLMs as Operating Systems.
[5] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications.
[6] Yan et al. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning.