acceptodds
Under review as a conference paper at ICLR 2027

ExperienceIndex: Artifact-Grounded Memory

Abstract

Knowledge‑intensive tasks in domains like law, science, and technology require answering many questions by reasoning about a shared corpus of artifacts (e.g., court cases, scientific literature, or software libraries). As humans interact with these corpora, they naturally accumulate experiential knowledge about artifacts, enabling them to quickly identify the complete set of relevant artifacts for each new task. However, existing AI agents lack appropriate memory solutions to build or reuse such *artifact-grounded experience*, leading to lower answer quality and higher online cost. Existing memory solutions extract and reuse information from prior task-solving traces, but they primarily focus on user preferences, factual attributes, or abstract reasoning patterns rather than persistent artifact‑specific knowledge. We introduce **ExperienceIndex**, a novel experience layer for AI agents that captures and reuses knowledge about artifacts based on prior reasoning traces. ExperienceIndex stores two complementary forms of experience: (i) single‑artifact experiences that summarize an artifact's contribution to prior tasks and (ii) artifact‑pair experiences that encode structural relationships discovered during past reasoning. Integrated as lightweight middleware, ExperienceIndex uses an experience retrieval mechanism to guide agents toward the complete set of relevant artifacts for new tasks, improving both answer quality and efficiency. Across diverse corpora and agentic solutions with different search frameworks, ExperienceIndex delivers consistent gains, raising answer quality by up to 11.0 points and reducing online dollar cost by up to 50.5%. We further demonstrate two benefits: (i) cross‑task generalization, where experiences accumulated from text‑to‑SQL tasks transfer to factoid QA tasks over the same artifact corpus, and (ii) teacher-student learning, where experiences from a stronger model enable a weaker model to reach comparable performance. These results highlight ExperienceIndex as a general, modular approach for enabling agents to build and leverage persistent, artifact‑grounded experience.

Primary area:
foundation or frontier models, including LLMs
Keywords:
memory, reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GWj7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an approach for capturing, retrieving, and reusing experience in solving tasks for agentic AI in which (similar to RAG) a corpus of external documents (called artifacts) is essential to solving those tasks. The work is motivated by enterprise application scenarios in which similar tasks are enacted repeatedly based on the same corpus, e.g. enterprise data. The main idea is to collect a) experience about which documents contribute which information for solving the tasks (called single-artifact experience) and b) experience about relationships between documents wrt. a specific task (called artifact-pair experience). The paper describes a middleware for handling experience in the descibed way, which can be used in combination with arbitrary LLMs and retrieval tools. The approach is evaluated with different kinds of tasks and datasets, demonstrating an increased answer quality and reduced LLM cost.

Critical strengths

The paper adresses a well-motivated and important practical problem and proposes an imporvement opportunity relevant for introducing genAI into enterprise environments.

The basic idea of using experiential knowledge within agentic AI is largely underexplored and this this paper makes a relevant contribution to this field of research.

Critical weaknesses

The main weakness of the paper is that it totally ignores decades of existing AI research on problem-solving based on experience and with this respect it tries to reinvent the wheel. Please take a look at the field of case-based reasoning (CBR), including seminal work by Aamodt & Plaza (1994) describing the CBR-Cycle, which is in fact instantiated by the paper without referring to it. Further, recent discussions on the intersection of CBR & LLMs as reflected in the annual International Conference on Case-Based Reasoning (ICCBR) and the papers published there will certainly provide valuable related work, not discussed in the submitted paper. The CBR-LLM manifesto from the CBR community could be helpful as well, see https://hal.science/hal-05006761 .

The paper claims that the observed improvements are based on the fact that the stored experience is related to the individual artifacts. However, experiences are built from 80% of each dataset and evaluated on the remaining 20% over the same corpus. The stored snippets and contribution summaries may contain the answer of the task, as shown with the Locomo-Dataset in Fig. 3 (the "yoga task"). There is no analysis of the overlap between experience and evaluation tasks. Thus, it is unclear if the quality improvement is due to the artifact experience or just a result of cashed answers to near-by questions.

Questions for authors
  • How can you be sure that both types of experience are relevant for the observed improvements? No ablation study is performed wrt. single-artifact vs. artifact-pair experiences.

  • How are combined similarities computed, i.e. similarity of task descriptions and similarities based on summaries?

  • What is the value of k and are there limits on the number of experiences injected?

  • How exactly are artifact-pair experiences represented. This is not explained?

  • Lightweight reflection is proposed but insufficiently described. Please provide more details. It remains unclear how it can "verify experience against the current artifacts, giving them priority over past experiences in the event of discrepancies caused by hallucinations, errors, or outdated information"

  • How is experience aggregated related to an artifact based on different tasks? What happens if over time, the experience-base grows and thus the effort for experience retrieval increases? What about forgetting or generalizing experience?

2 comments
AuthorsOpenReview ↗

Author response to Reviewer GWj7 (Part 1)

Thank you for the thoughtful review and feedback. We address your concerns and answer your questions below.

W1. [...] Please take a look at the field of case-based reasoning (CBR), including seminal work by Aamodt & Plaza (1994) describing the CBR-Cycle, which is in fact instantiated by the paper without referring to it. [...]

Thank you for bringing this line of work to our attention and we apologize for missing it. We have revised our paper to discuss ExperienceIndex’s relation to CBR in the related work (see the blue text in Section 2). We reproduce the text below.

Case-based reasoning. Case-based reasoning (CBR) is a longstanding class of techniques that reuse knowledge gained from past problems to help solve new ones (Aamodt & Plaza, 1994; Bach et al., 2025; Wiratunga et al., 2024). ExpIdx shares the underlying principle of the CBR cycle, accumulating and reusing knowledge from past problem-solving experiences. Our work focuses specifically on deriving and storing experiences about artifacts and their relationships, rather than organizing experiences around past problem cases. This artifact-grounded representation helps with experience reuse across different kinds of tasks sharing the same corpus, for example, transferring experiences learned from text-to-SQL (e.g., about table relationships) to factoid question answering over the same tables.

W2. [...] However, experiences are built from 80% of each dataset and evaluated on the remaining 20% over the same corpus. The stored snippets and contribution summaries may contain the answer of the task, as shown with the Locomo-Dataset in Fig. 3 (the "yoga task"). There is no analysis of the overlap between experience and evaluation tasks. Thus, it is unclear if the quality improvement is due to the artifact experience or just a result of cashed answers to near-by questions.

In our evaluation, we ensured that there was no task overlap between the experience tasks (used to construct the memory store) and the evaluation tasks (used to evaluate the agent). In other words, the same task was not present in both task sets.

Figure 4 in Appendix C.1 shows an example where artifact experience improves task solving quality. Here, the agent leverages artifact-pair experience to recognize that the Order_Items_d9a3 table can be joined with Products_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent did not know to consider Products_eafb and ultimately arrived at the wrong answer.

Q1. How can you be sure that both types of experience are relevant for the observed improvements? No ablation study is performed wrt. single-artifact vs. artifact-pair experiences.

Appendix C shows case studies where each type of experience was used to help an agent solve a task correctly that it was unable to do without the experience.

In Figure 3, the agent leverages single-artifact experience about session13.md on a related previous task (“How long has Jolene been doing yoga?”) to correctly answer the task at hand “Which year did Jolene start practicing yoga?” Without this single-artifact experience, the agent was unable to find the appropriate artifact that contained the answer.

In Figure 4, the agent leverages artifact-pair experience to recognize that the Order_Items_d9a3 table can be joined with Products_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent overlooked Products_eafb and ultimately arrived at the wrong answer.

In a revision, we will run a full ablation to study how each experience type contributes to the task solving accuracy and cost.

Q2. How are combined similarities computed, i.e. similarity of task descriptions and similarities based on summaries?

We compute the cosine similarity of the query embedding with respect to the past experience’s task description embedding and contribution summary embedding. We then combine them into one similarity measure by taking their mean.

Q3. What is the value of k and are there limits on the number of experiences injected?

We used k=5k = 5. Yes, kk is the upper bound on the number of experiences injected per artifact.

Q4. How exactly are artifact-pair experiences represented. This is not explained?

In our memory store database, they are represented as 3-element tuples: (artifact_id_1, artifact_id_2, relationship) where artifact_id_1 and artifact_id_2 are the IDs of the artifacts in the pair and relationship is a short natural language description of how the two artifacts are related.

AuthorsOpenReview ↗

Author response to Reviewer GWj7 (Part 2)

Q5. Lightweight reflection is proposed but insufficiently described. Please provide more details. It remains unclear how it can "verify experience against the current artifacts, giving them priority over past experiences in the event of discrepancies caused by hallucinations, errors, or outdated information"

When injecting prior experiences, we instruct the agent to always verify the experiences against the current artifacts. See Appendix B.2 for the prompt used.

An example of this reflection being used is in Figure 3 (Appendix C.1). The prior experience points to session13.md as a relevant artifact. Instead of simply responding by reasoning over the contribution summary, the agent checks the contents of session13.md to verify that the experience agrees before responding with its final answer.

We further demonstrate how an erroneous past experience leads to an incorrect answer when reflection is absent, versus a correct answer when reflection is applied.

  • Solved task: “What does Joanna do to remember happy memories?”
    • The model referenced the wrong file (session13.md) due to flawed reasoning, misinterpreting a transcript where Nate gives Joanna a stuffed animal for "good vibes" as a strategy for remembering happy memories. This resulted in the incorrect answer: "Keeps a stuffed animal."
  • New task: “What did Joanna receive from her brother that brought back childhood memories?”
    • Without reflection: The model blindly propagates the previous error. It uses the erroneous past experience without (1) assessing its actual relevance to "childhood memories" or (2) verifying factual correctness and freshness against current artifacts (such as whether Nate is actually Joanna's brother or if the information is outdated). This results in an incorrect answer.
    • With reflection: The model successfully (1) evaluates the available content to search for explicit mentions of "childhood memories" and (2) cross-checks past experiences against current artifacts (verifying familial relationships and other reasoning). This enables the system to discard the errors and yield the correct answer: "a handwritten letter."

Q6. Part 1: How is experience aggregated related to an artifact based on different tasks?

Each single-artifact recorded experience includes (i) a task description, which is the task from which this experience entry was derived, and (ii) a contribution summary, which describes how the artifact contributed to solving the task. See Section 3.1. Experience entries are not aggregated per artifact; we store each experience entry.

Q6. Part 2: What happens if over time, the experience-base grows and thus the effort for experience retrieval increases?

Our experience retrieval process is lightweight, ensuring that latency increases slowly as the memory store grows. This process consists of two lightweight steps: a top-kk vector search over past experiences (k=5k = 5 in our experiments), followed by an LLM-based relevance analysis of those top-kk experiences. For the top-kk vector search, widely adopted algorithms such as Hierarchical Navigable Small World (https://ieeexplore.ieee.org/abstract/document/8594636) offer a low time complexity of O(log⁡N)O(\log N) where NN represents the size of the vector database, while maintaining near-exact empirical accuracy. For the subsequent relevance analysis, because kk remains a small constant, the processing time stays stable. Consequently, even as stored experiences accumulate over time, the overall latency grows at a slow rate.

Q6. Part 3: What about forgetting or generalizing experience?

These are both additional features that can be added to the ExperienceIndex framework. For example, older experiences can be downweighted and offline analysis passes could aggregate past experiences into more general experiences about an artifact. We leave the exploration of these directions to future work.

Reviewer_LsAPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ExperienceIndex stores artifact-specific contributions and artifact-pair relationships extracted from prior reasoning traces, then retrieves them to guide new tasks over a shared corpus. Seven workloads and two solvers demonstrate answer-quality and online-cost benefits, including cross-task reuse.

Critical strengths
  1. Artifact identity provides a practical anchor for reusable experience. Incremental extraction from encountered artifacts avoids requiring full-corpus preprocessing and supports open-corpus use, a meaningful distinction from offline enrichment (Sections 2–3).
  2. Diverse workloads, search tools, and solvers support usefulness. Spider-to-MMQA transfer tests reuse across task types, while separate indexing and online costs make the trade-off assessable (Tables 2–4). Shared-corpus reuse is an intended feature of the setting.
Critical weaknesses
  1. The ablations do not isolate the central memory design. Section 5 examines reflection, memory size, and model choices, but does not remove artifact-pair links or compare artifact-grounded versus task-grounded memories under matched retrieval and injection. Answer-score gains therefore do not establish whether pair experiences improve evidence coverage or which representational choices drive the gains.
  2. Applicability to small-parameter models remains unclear. Table 5 varies the relevance model, and Table 4 tests stronger-to-weaker experience transfer, but neither establishes robustness across solver sizes. Smaller solvers may differ in their ability to use retrieved experience and reject misleading memories, which matters for the method’s cost-saving potential.
Questions for authors
  1. On one multi-hop workload, can you compare full memory, single-artifact-only memory, and task/trace memory using the same traces, relevance model, and injection budget? Report relevant-artifact recall alongside answer quality and online cost.
  2. With the memory store and relevance model fixed, does the method improve a small-parameter open-weight solver over its no-memory baseline? Reporting quality and total online cost, including retrieval and relevance filtering, would clarify whether the benefits extend to smaller models.
1 comment
AuthorsOpenReview ↗

Author response to Reviewer LsAP

Thank you for the thoughtful review and feedback. We address your concerns and answer your questions below.

W1. Part 1: The ablations do not isolate the central memory design. Section 5 examines reflection, memory size, and model choices, but does not remove artifact-pair links or compare artifact-grounded versus task-grounded memories under matched retrieval and injection.

We compare ExperienceIndex against ReasoningBank, which is a form of task-grounded memory, see Table 2 (Section 4.2). Compared with ReasoningBank, ExperienceIndex has a comparable indexing cost and a storage footprint of the same order of magnitude. Crucially, ExperienceIndex also improves average answer score by 5.01% and reduces average online cost by 19.8%.

Regarding artifact-pair experiences and ablations, see below.

W1. Part 2: Answer-score gains therefore do not establish whether pair experiences improve evidence coverage or which representational choices drive the gains.

Figure 4 in Appendix C.1 shows how artifact-pair experience helps an agent correctly solve a task it would otherwise fail. Specifically, the agent leverages artifact-pair experience to recognize that the Order_Items_d9a3 table can be joined with Products_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent overlooked Products_eafb and ultimately arrived at the wrong answer.

In a revision, we will run a full ablation to study how each experience type contributes to the task solving accuracy and cost.

W2. Applicability to small-parameter models remains unclear. Table 5 varies the relevance model, and Table 4 tests stronger-to-weaker experience transfer, but neither establishes robustness across solver sizes. Smaller solvers may differ in their ability to use retrieved experience and reject misleading memories, which matters for the method’s cost-saving potential.

We evaluated ExperienceIndex using two solver models, GPT-5-Mini and Gemini-3-Flash; the former represents a smaller model compared to Gemini-3-Flash. Our results in Table 2 show that the smaller model (GPT-5-Mini) also brings accuracy gains and cost savings over the baselines.

In a revision, we will also evaluate ExperienceIndex on an even smaller open source model. Given the performance of GPT-5-Mini versus Gemini-3-Flash, we expect the even smaller open source model to also have gains over the baseline.

Q1. On one multi-hop workload, can you compare full memory, single-artifact-only memory, and task/trace memory using the same traces, relevance model, and injection budget? Report relevant-artifact recall alongside answer quality and online cost.

Please see our response to W1 Part 2.

Q2. With the memory store and relevance model fixed, does the method improve a small-parameter open-weight solver over its no-memory baseline? Reporting quality and total online cost, including retrieval and relevance filtering, would clarify whether the benefits extend to smaller models.

Please see our response to W2.

Reviewer_CeDHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ExperienceIndex is an agent memory layer that anchors experience on corpus artifacts rather than on user profiles or abstract strategies. From prior reasoning traces, an LLM extracts single-artifact experiences (task, contribution summary, snippets) and artifact-pair relations. ExperienceIndex acts as middleware: it injects experiences into the prompt and rewrites search outputs by pruning irrelevant artifacts and adding relevant neighbors. Across seven datasets with Gemini-3-Flash and GPT-5-Mini, it reports up to +11.0 points and up to 50.5% lower online cost against no memory, ReasoningBank and EnrichIndex, plus cross-task and teacher–student transfer.

Critical strengths
  1. Anchoring memory on artifacts fills a clear gap between profile-based memory (Mem0) and strategy-level memory (ReasoningBank).
  2. The middleware is genuinely framework-agnostic, shown with file-system tools, vector search and ScholarQA's pipeline.
  3. Coverage is broad, with seven datasets spanning conversations, code, literature and relational tables.
  4. Indexing is ≈115× cheaper with ≈73× less storage than EnrichIndex, and needs no full-corpus access.
Critical weaknesses
  1. About 100 test tasks per dataset with no confidence intervals, so many reported gains (some under one point) are within noise.
  2. ExpIdx is not best in several cells and raises online cost with GPT-5-Mini on four of seven datasets.
  3. The headline aggregates (3.9%, 20.8%) only hold with SQA excluded, which the text does not state.
  4. A random 80/20 split over a shared corpus allows near-duplicate questions and possible post-fix code leakage on SWE-bench.
  5. There is no component ablation and no simple past-question-plus-snippet retrieval baseline, all analyses use Locomo only.
Questions for authors

Please address these following questions:

  1. Can authors report bootstrap CIs and state how SQA enters the aggregates?
  2. What fraction of test tasks have their gold evidence or a near-duplicate question already in memory, and how do results look without them?
  3. Was the SWE-bench-Verified split chronological within each repository?
  4. Can authors ablate artifact-pair experience and the LLM relevance filter, and add a simple trace-retrieval baseline?
  5. Does the Gemini-3-Flash online cost include the GPT-5-Mini relevance-analysis calls and embedding costs? Please clarify.
1 comment
AuthorsOpenReview ↗

Author response to Reviewer CeDH

Thank you for the thoughtful review and feedback. We address your concerns and answer your questions below. Our responses appear in the order that the questions were given to us and grouped when applicable.

W1. About 100 test tasks per dataset with no confidence intervals, so many reported gains (some under one point) are within noise.

W3. The headline aggregates (3.9%, 20.8%) only hold with SQA excluded, which the text does not state.

Q1. Can authors report bootstrap CIs and state how SQA enters the aggregates?

The main takeaway from our work is that ExpIdx achieves accuracy and cost improvements across a diverse range of datasets, covering multiple modalities (files, documents, and tables) and task types (Table 1). ExpIdx achieves accuracy gains up to 11.0 points and reduces online dollar cost by up to 50.5%.

The 3.9% and 20.8% averages are over the next best baseline, which is EnrichIndex. EnrichIndex is not evaluated on SQA, since it requires offline access to the full Semantic Scholar corpus (238 million+ papers). We discuss this in the Table 2 caption.

W2. ExpIdx is not best in several cells and raises online cost with GPT-5-Mini on four of seven datasets.

ExpIdx achieves at least either the best accuracy or best cost in each dataset. Averaged across both evaluation models, ExpIdx achieves the highest answer score and lowest online cost over the next best baseline.

W4. A random 80/20 split over a shared corpus allows near-duplicate questions and possible post-fix code leakage on SWE-bench.

Q2. What fraction of test tasks have their gold evidence or a near-duplicate question already in memory, and how do results look without them?

In our evaluation, we ensured that there was no task overlap between the experience tasks (used to construct the memory store) and the evaluation tasks (used to evaluate the agent). In other words, the same task was not present in both task sets.

W5. There is no component ablation and no simple past-question-plus-snippet retrieval baseline, all analyses use Locomo only.

Appendix C shows case studies where each type of experience was used to help an agent solve a task correctly that it was unable to do without the experience.

In Figure 3, the agent leverages single-artifact experience about session13.md on a related previous task (“How long has Jolene been doing yoga?”) to correctly answer the task at hand “Which year did Jolene start practicing yoga?” Without this single-artifact experience, the agent was unable to find the appropriate artifact that contained the answer.

In Figure 4, the agent leverages artifact-pair experience to recognize that the Order_Items_d9a3 table can be joined with Products_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent overlooked Products_eafb and ultimately arrived at the wrong answer.

In a revision, we will run a full ablation to study how each experience type contributes to the task solving accuracy and cost.

Q3. Was the SWE-bench-Verified split chronological within each repository?

Yes.

Q4. Can authors ablate artifact-pair experience and the LLM relevance filter, and add a simple trace-retrieval baseline?

Regarding ablations, see W5.

We compare ExperienceIndex against ReasoningBank, which is a form of trace-based memory that extracts generalizable insights from past traces. Compared with ReasoningBank, ExperienceIndex has a comparable indexing cost and a storage footprint of the same order of magnitude. Crucially, ExperienceIndex also improves average answer score by 5.01% and reduces average online cost by 19.8%. See Table 2 (Section 4.2).

Q5. Does the Gemini-3-Flash online cost include the GPT-5-Mini relevance-analysis calls and embedding costs? Please clarify.

Yes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.