acceptodds
Under review as a conference paper at ICLR 2027

LongMemEval-V2: Long-Term Agent Memory Evaluation Toward Experienced Colleagues

17 views

Abstract

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. These questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. As initial baselines for this challenging setting, we propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%). Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems that turn accumulated agent trajectories into reusable environment experience.

Primary area:
datasets and benchmarks
Keywords:
long-term memory, agent memory, retrieval-augmented generation, web agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_AmEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongMemEval-V2, a benchmark designed to evaluate whether agents can accumulate and use reusable environment-specific experience from large collections of historical web interaction trajectories. It contains 451 questions covering five abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. The questions are evaluated against histories of up to approximately 500 trajectories, or 115M tokens. Under the Context Gathering protocol, a memory system first processes the historical trajectories and then retrieves evidence for a given query, which a fixed reader uses to generate the final answer. The paper also introduces two baselines, AgentRunbook-R and AgentRunbook-C. AgentRunbook-C achieves accuracies of 74.9% on Small and 70.1% on Medium, offering a reasonable trade-off between accuracy and query latency.

Critical strengths
  1. Meaningful research problem. Long-term agent memory should capture not only user preferences but also reusable, environment-specific operational experience. The paper clearly identifies and focuses on this problem.
  2. Large-scale benchmark. LME-V2 includes 25.6M–114.8M tokens of historical data and links questions to multiple web-agent trajectories. Compared with benchmarks based on conversational histories, this setup better reflects the needs of long-running web and enterprise agents.
  3. Clear presentation. The paper is well written and easy to follow.
Critical weaknesses
  1. Limited benchmark novelty. The distinction from existing agent-memory benchmarks is not sufficiently clear. For example, AMA-Bench already evaluates long-term memory over agent trajectories and includes a memory system based on causal graphs and tool-augmented retrieval. The paper should make a clearer comparison and explain which unique capabilities LME-V2 measures and why existing benchmarks cannot adequately evaluate them.
  2. Limited domain coverage. The benchmark is restricted to customized web-browsing environments and a small set of websites in WebArena and WorkArena. It does not cover other agent settings, such as operating-system agents, embodied agents, or coding agents.
  3. Limited diversity in memory approaches. Both AgentRunbook baselines rely on retrieval and tool use. The paper does not explore trainable memory, memory distillation, or continual memory updates. It also lacks direct experimental comparisons with established agent-memory systems such as MemGPT, Mem0, and Memory-R1; these systems are only briefly discussed in the related work.
  4. Some results need clarification. The introduction reports 40.1% accuracy for the basic query-to-slice RAG baseline, whereas Table 2 reports 42.8% on Small and 38.1% on Medium, giving an unweighted average of 40.45%. The paper should clarify how the reported figure is calculated. In addition, the mapping between the five memory abilities and the category-level results in Table 2 is unclear, particularly for premise-awareness questions.
Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Thank you for the review

Thank you for recognizing the importance of the problem and our effort to build a large benchmark. We address your concerns below.

Benchmark novelty

AMA-Bench [5] is concurrent work that focuses on memory within a task episode. It tests recall, causal inference, state updates, and state abstraction. LME-V2 asks what agents learn from 100-500 past trajectories, with explicit tests of reusable workflows, environment gotchas, and wrong premises. This adds the challenge of finding and combining relevant experience among many unrelated tasks. We will make these differences clearer.

Domain coverage

We agree that broader domains would be valuable. We chose customized websites because models cannot reliably answer questions about their specific details from prior knowledge alone (Figure 4). We also explored coding tasks, but found that models often already memorized the repositories, making it harder to construct questions that require memory of past experience. Extending the benchmark to coding, OS, and embodied agents is an important direction.

Diversity of memory approaches

Our main contribution is the benchmark. RAG pipelines and coding-agent memory provide two practical starting points, and AgentRunbook shows how each can be improved through memory design. The table summarizes differences in what existing systems preserve. For example, keeping only strategy notes can lose details needed for questions about specific UI fields.

SystemState detailsState transitionsProcedures / hintsDomainAgentic retrieval
ReasoningBank [1]✗✗✓Web, code✗
Mem0 [2]✓✓✗Dialogue✗
A-MEM [3]✓✗✗Dialogue✗
MemGPT / Letta V1 [4]✓✓✓Chat, documents✓
AMA-Agent [5]✓✓✗Mixed agents✓
AgentRunbook-R✓✓✓Web✗
AgentRunbook-C✓✓✓Web✓

Trainable memory, distillation, and continual updates are important directions for this benchmark. Memory-R1 [6], for example, learns memory operations through reinforcement learning. Our Insert/Query interface allows such approaches, and we hope LME-V2 will encourage their evaluation.

Result clarification

Thank you for spotting the inconsistency. The displayed Table 2 values average to 40.45%: (42.8% + 38.1%) / 2. We will correct the inconsistent number in the introduction and explain the aggregation.

As shown in Figure 2, premise-awareness questions are wrong-premise variants of static, dynamic, and workflow questions. Table 2 includes each variant in its corresponding category:

Table 2 columnQuestions included
StaticStatic recall, including wrong-premise variants
DynamicDynamic state tracking, including wrong-premise variants
WorkflowWorkflow knowledge, including wrong-premise variants
GotchasEnvironment gotchas

We will make this mapping explicit in the Table 2 caption.

References

[1] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.

[2] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.

[3] Xu et al. A-MEM: Agentic Memory for LLM Agents.

[4] Packer et al. MemGPT: Towards LLMs as Operating Systems.

[5] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications.

[6] Yan et al. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning.

Thank you for the rebuttal. The response partially addresses my concerns. I will keep my score unchanged for now and discuss the dataset’s novelty, practical value, and rigor with the other reviewers in the next phase.

Reviewer_xbWaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LongMemEval-V2 (LME-V2), a benchmark that evaluates whether memory systems can help web agents become “experienced colleagues” in customized environments. It contains 451 manually curated questions covering five memory abilities. Each question is paired with multimodal trajectory haystacks of up to ~500 trajectories (~115M tokens). The authors also propose two baselines: AgentRunbook-R, a multi-pool RAG pipeline, and AgentRunbook-C, a Codex-based memory module with workflow instructions, manifests, and helper scripts. AgentRunbook-C achieves the best accuracy (72.5%), outperforming vanilla Codex with lower latency.

Critical strengths
  1. The hybrid human–AI annotation pipeline is a practical way to build a benchmark at this scale. Codex proposes candidate answer-bearing trajectories using metadata and screenshot evidence, and human experts curate the seed set, resolve ambiguous cases, and verify the trajectories included in the final core haystack (§3.2, Appendix A.3). This removes the labeling bottleneck while keeping the multi-hop evidence mapping reliable.
  2. Framing memory as what makes an agent an “experienced colleague” is a meaningful step beyond retrieval-style QA over long documents or chat histories. The benchmark evaluates whether a memory system internalizes environment-specific experience, such as workflows, state dynamics, and recurring gotchas, rather than whether it can locate a single fact. I especially appreciate this contribution.
  3. The scale and modality of the haystacks are impressive. Each question comes with up to ~500 trajectories and ~115M tokens of history, and every state includes both a screenshot and an accessibility tree (Table 1, Figure 3, Appendix A.1). This realistically reproduces the noisy, multimodal context that a real web agent would accumulate.
Critical weaknesses
  1. The haystacks are constructed artificially rather than accumulated naturally. To reach the target scale, the authors select a minimal answer core by solving an assignment problem, insert filler trajectories ranked by diversity and difficulty, control the success/failure balance toward a 1:1 ratio, and then shuffle the order (Appendix A.4). This does not reflect how an agent actually accumulates and forgets experience while interacting with an environment over time. The authors acknowledge the offline setting in Appendix E.1, but the effect of the haystack composition itself on the results should also have been discussed.
  2. The benchmark does not validate end-to-end task success. The context-gathering formulation only measures whether the memory system returns context from which a fixed reader can answer the question (§3.3). It remains unclear whether higher LME-V2 accuracy translates into better action planning and higher task success for a downstream agent. The authors describe this as an intentional design choice (Appendix E.1), but even a small experiment on a subset of WebArena/WorkArena tasks would have greatly strengthened the claim that LME-V2 measures what matters for agents.
  3. The taxonomy of the five memory abilities lacks theoretical grounding. Static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness are defined only from the intuition of what an experienced colleague would internalize (§3.1). The paper does not connect this taxonomy to cognitive science (e.g., episodic, semantic, and procedural memory) or to prior agent memory literature. It also does not show that the categories are clearly separable; for example, dynamic and workflow questions both appear to require synthesizing many states within a trajectory (§3.2). Without this justification, the validity of the per-category analysis is weakened.
Questions for authors

Does higher LME-V2 accuracy translate into higher end-to-end task success when the memory is provided to a web agent? What is the estimated recall of the Codex-assisted answer-trajectory labeling? If it misses answer-bearing trajectories, some medium-tier fillers may unintentionally contain answers (Appendix A.4).

1 comment
AuthorsOpenReview ↗

Thank you for the review

Thank you for recognizing the dataset’s scale, our annotation effort, and the idea of evaluating agents as experienced colleagues. We address your concerns below.

Assembled vs. natural context

We assemble the histories to control how often the relevant evidence appears and to include both successful and failed attempts. This tests whether memory systems can find useful evidence in a long history and learn from failures. The system still receives trajectories one at a time and retrieves information when asked a question, as it would in deployment. We will explain how these choices affect the evaluation.

End-to-end success

We use a fixed reader for two reasons. First, task success depends on model planning and action execution capabilities as well as memory. Using the same reader makes memory systems easier to compare. Second, a successful task may never test a gotcha or a wrong premise. Our questions test these abilities directly. Testing whether higher LME-V2 scores predict better task success is an important next step.

Ad-hoc taxonomy

We agree that it would help to connect our five abilities to episodic, semantic, and procedural memory. We will add this discussion. The categories ask different questions: dynamic questions ask what changed after an action, while workflow questions ask how to complete a task. They can use the same trajectory without testing the same knowledge.

Recall of answer-trajectory labeling

We manually check that the selected trajectories provide enough evidence to answer each question (Appendix A.3-A.4). We do not report how many supporting trajectories were missed. If a missed trajectory is included as a filler, it gives the system extra evidence and can make retrieval easier. We will clarify this in the revision.

Reviewer_1AiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper claimed a new argument about evaluating agentic memory that "a high-quality memory makes an agent an experienced colleague in a specialized environment". They then defined 5 key abilities that an experienced colleague should have and evaluated how memory agents/systems behavior on these 5 core dimensions. To do so, they proposed LongMemEval-V2, a multimodal web-agent memory benchmark of 451 human-curated questions paired with massive trajectory histories up to 500 trajectories / 115M tokens. On this benchmark, they evaluated two internal baseline families (Dense Vector RAG vs File-system Coding agents) including their proposed AgentRunBook-R and AgentRunBook-C. Results shows the benchmark they proposed is challenging for current SOTA agent with various memory mechanisms.

Critical strengths

The proposed benchmark covers 4 tested agentic domains, and the authors collect more than 1500 long agent trajectories. All these efforts make the benchmark a good resource for future agent memory research. The paper also defines 5 core abilities that a good memory system should have, setting a good standard to follow and inspiring future work.

The proposed memory method, AgentRunBook, is simple but effective. AgentRunBook-R has a novel multi-pool RAG structural design. AgentRunBook-C shows a good demo on building scaffold coding-agent memory.

The pilot studies also proves the soundness and high-quality of the benchmark.

Critical weaknesses

I have concern over the baseline comparison that the comparisons have confounded backbones. For example, the RAG methods use Qwen3.5-9B with pure dense vector search (Qwen3-Embedding-8B), whereas all Coding Agent methods use GPT-5.4-mini (xhigh) and exact lexical file search (ripgrep/find). The claimed advantage of coding-agent memory is confounded with stronger LLM reasoning and different search mechanism (and exact keyword matching might be critical for UI button/field names). Running AgentRunbook-R with GPT-5.4-mini or add lexical BM25 to RAG might be helpful to decouple the comparison factors and offer a more sound evaluation.

I also have a concern over the small baseline set. The authors hypothesize that some conversational memory system are out of scope which is understandable, but there are still external memory agent such as AMA-Agent or Letta that align better. Trying more baselines even with LME-V2-Small (if cost is a reason) will improve the paper.

My last concern is about the potential metadata leakage. As shown in Table 10, AgentRunbook-R’s query-generation prompt explicitly passes golden question_type into model. Also in Table 11, AgentRunbook-C’s question includes metadata too. Telling the memory controller upfront whether a question is workflow, static, or abstention leaks benchmark annotations and may provide evaluation hint to model. Also it makes the evaluation less realistic as a real deployed memory module would never have that at query time.

Questions for authors

In sec 3.3, the formal evaluation API is strictly ci=Query(qi)c_i = Query(q_i), but in AgentRunbook-R and AgentRunbook-C, the annotated question type/metadata also get leakage. Any reason for doing this design?

2 comments
AuthorsOpenReview ↗

Thank you for the review

Thank you for recognizing the benchmark’s value and AgentRunbook’s effectiveness. We address your concerns below.

Comparison across families

We compare two common deployment choices: RAG over extracted pools and coding-agent search over trajectory files. Their representations and search mechanisms are coupled, while Table 2’s within-family ablations examine individual design choices. In our pilot, Qwen3.5-9B and GPT-5.4-mini performed similarly as AgentRunbook-R’s query generator. We will report this controller ablation with retrieval settings and query budget fixed in the revision.

Small baseline set

The table compares the knowledge represented by existing systems and their retrieval mechanisms.

SystemState detailsState transitionsProcedures / hintsDomainAgentic retrieval
ReasoningBank [1]✗✗✓Web, code✗
Mem0 [2]✓✓✗Dialogue✗
A-MEM [3]✓✗✗Dialogue✗
MemGPT / Letta V1 [4]✓✓✓Chat, documents✓
AMA-Agent [5]✓✓✗Mixed agents✓
AgentRunbook-R✓✓✓Web✗
AgentRunbook-C✓✓✓Web✓

Abstraction into strategy notes can discard UI details needed for static-state questions. This motivates preserving both raw evidence and reusable knowledge. We will include comparisons with relevant external systems such as Letta and AMA-Agent in the final revision.

Metadata leakage

Thank you for catching this. The reported results were obtained after the bug was fixed: no method receives question-type labels or original task goals as query metadata. Tables 10-11 contain outdated prompt descriptions, which we will correct.

References

[1] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.

[2] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.

[3] Xu et al. A-MEM: Agentic Memory for LLM Agents.

[4] Packer et al. MemGPT: Towards LLMs as Operating Systems.

[5] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications.

Thanks for the responses. After carefully reviewing the response, I believe the paper could be improved a lot by another round of revision, especially if authors can add the discussed ablations (e.g., controller ablation with different retrieval setups, query budgets, etc) to further prove the soundness of the evaluations. Therefore I lean to maintain my current score.

Reviewer_Vn8POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces 451 questions about environment knowledge learned from web-agent trajectories, with the Medium history tier averaging about 115M tokens. It also proposes RAG and coding-agent memory baselines; AgentRunbook-C reaches 72.5% average accuracy across two history sizes.

Critical strengths
  1. The five ability categories cover workflows, state transitions, and misleading premises, extending memory evaluation beyond fact recall (Section 3.1).
  2. The dataset has careful human checks, sanitized task goals, and validated grading. The oracle and no-context studies help establish what the questions require (Appendices A-B).
  3. AgentRunbook-C has higher reported accuracy than Codex with the same controller, while reducing query latency from 177 to 108 seconds on Small and 186 to 140 seconds on Medium (Table 2).
Critical weaknesses
  1. Table 10's query template includes question type and original goals linked to the question. It is unclear whether these fields are populated during evaluation or shared across methods; if used, they could provide retrieval hints beyond the question itself.
  2. The cross-family comparison changes both memory design and controller model: Qwen3.5-9B for RAG versus GPT-5.4-mini for coding agents (Table 2). The reader swap in Appendix D.1 does not isolate these effects, although the comparison with Codex uses the same controller.
Questions for authors
  1. Are the question-type and original-goal fields populated at test time, and which methods receive them? If used, do the gains hold with only the question text and image?
  2. Does the coding-agent advantage remain when the RAG controller also uses GPT-5.4-mini under a comparable query budget?
  3. The related-work discussion could include interactive benchmarks such as MemGUI-Bench, MobileWorld, and OSWorld 2.0, distinguishing task-completion evaluation from trajectory-based QA.
1 comment
AuthorsOpenReview ↗

Thank you for the review

Thank you for recognizing the benchmark’s capability coverage and quality, and AgentRunbook’s practical performance. We address your concerns below.

Metadata leakage

Thank you for catching this. The reported results were obtained after the bug was fixed: no method receives question-type labels or original task goals as query metadata. Tables 10-11 contain outdated prompt descriptions, which we will correct.

Cross-family comparison

Qwen3.5-9B and GPT-5.4-mini performed similarly as AgentRunbook-R’s query generator in our pilot study, so we chose the cheaper Qwen model. We will report this controller ablation with the retrieval pipeline and query budget held fixed in the revision.

Extended related work

We will add MemGUI-Bench, MobileWorld, and OSWorld 2.0 to the related work. Task-completion evaluation is valuable, especially across dependent sessions as in MemoryArena [1]. Completion scores also reflect the agent’s planning and execution, and can saturate as models improve. LME-V2 complements these evaluations by fixing the reader and directly testing the environment knowledge recovered from past trajectories.

References

[1] He et al. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks.

Reviewer_4afrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LongMemEval-V2, a benchmark designed to directly evaluate how effectively memory systems can internalize environment-specific experiences. While existing memory benchmarks primarily focus on user histories, short traces, or downstream task success, this work identifies five core memory abilities and manually constructs and annotates trajectories to evaluate these abilities. Furthermore, by using a fixed reader model, the benchmark separates memory system performance from the reasoning capabilities, providing a more isolated evaluation framework. Finally, the authors propose two memory modules, AgentRunbook-R, a RAG-based approach, and AgentRunbook-C, a coding agent-based approach, offering insights into the design of effective memory systems.

Critical strengths

S1. Well-defined memory capabilities

  • The paper identifies five core memory abilities that have been largely overlooked by existing benchmarks, providing a more comprehensive framework for evaluating agent memory systems.

S2. High-quality benchmark construction

  • The authors put considerable effort into constructing a high-quality benchmark. Human experts manually verify annotations for the questions, and the authors carefully construct haystacks to minimize the number of answer-bearing trajectories, thereby controlling the difficulty of evidence retrieval. Additionally, the use of a fixed reader model enables a more isolated evaluation of memory system performance.
Critical weaknesses

W1. Limited comparison of existing memory systems

  • AgentRunbook-R and AgentRunbook-C appear to effectively integrate existing approaches, but their methodological novelty seems somewhat limited.
  • Therefore, broader comparisons with existing memory methods would help better justify the proposed designs and provide a more comprehensive understanding of limitations of different memory systems.
  • For example, analyzing which types of information existing methods struggle to preserve or retrieve would provide stronger motivation for the design choices of AgentRunbook.

W2. Insufficient analysis of the proposed methods

  • AgentRunbook-R and AgentRunbook-C differ not only in how they store memories but also in how they retrieve information.
  • However, the paper primarily compares the two methods in terms of accuracy and latency, making it difficult to disentangle the individual contributions of memory representation and retrieval mechanisms.

W3. Insufficient failure analysis across question types

  • A more detailed analysis of why models fail on different question types would be valuable. For example, even when oracle trajectories are provided, performance on gotcha questions remains relatively low, but the underlying reasons are not clearly explained.
  • Beyond reporting performance differences, a more fine-grained failure analysis would help identify the specific limitations of current system.
Questions for authors

Q1. Are there any potential side effects associated with inserting trajectories sequentially? For example, do trajectories exhibit temporal dependencies, and if so, how might these dependencies affect memory construction, retrieval, or evaluation?

1 comment
AuthorsOpenReview ↗

Thank you for the review

Thank you for recognizing our ability coverage and the care taken in constructing the benchmark. We address your concerns below.

Comparison with existing memory systems

Our main contribution is the LME-V2 benchmark. AgentRunbook provides practical baselines, with the following differences from representative memory systems:

SystemState detailsState transitionsProcedures / hintsDomainAgentic retrieval
ReasoningBank [1]✗✗✓Web, code✗
Mem0 [2]✓✓✗Dialogue✗
A-MEM [3]✓✗✗Dialogue✗
MemGPT / Letta V1 [4]✓✓✓Chat, documents✓
AMA-Agent [5]✓✓✗Mixed agents✓
AgentRunbook-R✓✓✓Web✗
AgentRunbook-C✓✓✓Web✓

AWM and ReasoningBank emphasize reusable workflows and strategies. LME-V2 also tests fine-grained details, such as whether an incident form contains a Resolve button (Figure 1). We will expand the related work around these distinctions.

Analysis of proposed methods

We study two common deployment choices: RAG over extracted memory pools and coding-agent search over trajectories and summaries. Their representations and retrieval mechanisms are coupled. We therefore use the within-family ablations in Table 2 to examine design choices: on Small, removing raw slices lowers AgentRunbook-R from 58.6% to 42.3%, and removing workflow guidance lowers AgentRunbook-C from 74.9% to 70.1%.

Failure analysis across question types

We will expand Appendix D.2 with analysis by question type. Even with oracle trajectories, relevant evidence can be incidental to the original task, and gotchas can require inferring rules behind trial-and-error actions. For example, the filter gotcha in Figure 1 requires recognizing that the default comparison is “≥”. This motivates extracting notes that surface implicit knowledge and inspecting targeted states for supporting evidence.

Side effects

Each trajectory starts from a snapshot of the same environment, and its internal state sequence remains ordered. The questions do not depend on a global chronology across trajectories or small inconsistencies between snapshots. Simulating sessions in a shared, continuously evolving world is an important future direction.

References

[1] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.

[2] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.

[3] Xu et al. A-MEM: Agentic Memory for LLM Agents.

[4] Packer et al. MemGPT: Towards LLMs as Operating Systems.

[5] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you for your insightful reviews. We are encouraged that you recognize these strengths:

  • An important focus on reusable environment knowledge (xbWa, AmEu).
  • Useful memory abilities and careful benchmark validation (4afr, Vn8P, 1AiY).
  • Large histories of agent experience (1AiY, xbWa, AmEu).
  • Effective and practical memory baselines (Vn8P, 1AiY).

We address the individual concerns in our replies and highlight two common points below.

Why QA rather than task completion?

Our goal is to test what memory systems retain and can recover. We use QA for two reasons. First, fixing the reader makes memory systems easier to compare, while task success also depends on planning and action execution. Second, finishing a task may never test whether an agent recognizes a gotcha or a wrong premise. Our questions test these abilities directly.

Why these memory baselines?

LME-V2 is our main contribution. We include RAG and coding-agent memory as two practical ways to use past trajectories, with ablations to study their design choices. The table below explains why memory representation matters: a system that keeps only high-level strategies may lose the UI details needed for some of our questions.

SystemState detailsState transitionsProcedures / hintsDomainAgentic retrieval
ReasoningBank [1]✗✗✓Web, code✗
Mem0 [2]✓✓✗Dialogue✗
A-MEM [3]✓✗✗Dialogue✗
MemGPT / Letta V1 [4]✓✓✓Chat, documents✓
AMA-Agent [5]✓✓✗Mixed agents✓
AgentRunbook-R✓✓✓Web✗
AgentRunbook-C✓✓✓Web✓

We will expand comparisons with external systems and clarify these design differences in the revision. We hope LME-V2 will provide a useful testbed for a broader range of memory methods.

References

[1] Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.

[2] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.

[3] Xu et al. A-MEM: Agentic Memory for LLM Agents.

[4] Packer et al. MemGPT: Towards LLMs as Operating Systems.

[5] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications.

Sincerely,

Paper 46773 authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.