- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an approach for capturing, retrieving, and reusing experience in solving tasks for agentic AI in which (similar to RAG) a corpus of external documents (called artifacts) is essential to solving those tasks. The work is motivated by enterprise application scenarios in which similar tasks are enacted repeatedly based on the same corpus, e.g. enterprise data. The main idea is to collect a) experience about which documents contribute which information for solving the tasks (called single-artifact experience) and b) experience about relationships between documents wrt. a specific task (called artifact-pair experience). The paper describes a middleware for handling experience in the descibed way, which can be used in combination with arbitrary LLMs and retrieval tools. The approach is evaluated with different kinds of tasks and datasets, demonstrating an increased answer quality and reduced LLM cost.
Critical strengths
The paper adresses a well-motivated and important practical problem and proposes an imporvement opportunity relevant for introducing genAI into enterprise environments.
The basic idea of using experiential knowledge within agentic AI is largely underexplored and this this paper makes a relevant contribution to this field of research.
Critical weaknesses
The main weakness of the paper is that it totally ignores decades of existing AI research on problem-solving based on experience and with this respect it tries to reinvent the wheel. Please take a look at the field of case-based reasoning (CBR), including seminal work by Aamodt & Plaza (1994) describing the CBR-Cycle, which is in fact instantiated by the paper without referring to it. Further, recent discussions on the intersection of CBR & LLMs as reflected in the annual International Conference on Case-Based Reasoning (ICCBR) and the papers published there will certainly provide valuable related work, not discussed in the submitted paper. The CBR-LLM manifesto from the CBR community could be helpful as well, see https://hal.science/hal-05006761 .
The paper claims that the observed improvements are based on the fact that the stored experience is related to the individual artifacts. However, experiences are built from 80% of each dataset and evaluated on the remaining 20% over the same corpus. The stored snippets and contribution summaries may contain the answer of the task, as shown with the Locomo-Dataset in Fig. 3 (the "yoga task"). There is no analysis of the overlap between experience and evaluation tasks. Thus, it is unclear if the quality improvement is due to the artifact experience or just a result of cashed answers to near-by questions.
Questions for authors
-
How can you be sure that both types of experience are relevant for the observed improvements? No ablation study is performed wrt. single-artifact vs. artifact-pair experiences.
-
How are combined similarities computed, i.e. similarity of task descriptions and similarities based on summaries?
-
What is the value of k and are there limits on the number of experiences injected?
-
How exactly are artifact-pair experiences represented. This is not explained?
-
Lightweight reflection is proposed but insufficiently described. Please provide more details. It remains unclear how it can "verify experience against the current artifacts, giving them priority over past experiences in the event of discrepancies caused by hallucinations, errors, or outdated information"
-
How is experience aggregated related to an artifact based on different tasks? What happens if over time, the experience-base grows and thus the effort for experience retrieval increases? What about forgetting or generalizing experience?
2 comments
Author response to Reviewer GWj7 (Part 2)
Q5. Lightweight reflection is proposed but insufficiently described. Please provide more details. It remains unclear how it can "verify experience against the current artifacts, giving them priority over past experiences in the event of discrepancies caused by hallucinations, errors, or outdated information"
When injecting prior experiences, we instruct the agent to always verify the experiences against the current artifacts. See Appendix B.2 for the prompt used.
An example of this reflection being used is in Figure 3 (Appendix C.1). The prior experience points to session13.md as a relevant artifact. Instead of simply responding by reasoning over the contribution summary, the agent checks the contents of session13.md to verify that the experience agrees before responding with its final answer.
We further demonstrate how an erroneous past experience leads to an incorrect answer when reflection is absent, versus a correct answer when reflection is applied.
- Solved task: “What does Joanna do to remember happy memories?”
- The model referenced the wrong file (
session13.md) due to flawed reasoning, misinterpreting a transcript where Nate gives Joanna a stuffed animal for "good vibes" as a strategy for remembering happy memories. This resulted in the incorrect answer: "Keeps a stuffed animal."
- The model referenced the wrong file (
- New task: “What did Joanna receive from her brother that brought back childhood memories?”
- Without reflection: The model blindly propagates the previous error. It uses the erroneous past experience without (1) assessing its actual relevance to "childhood memories" or (2) verifying factual correctness and freshness against current artifacts (such as whether Nate is actually Joanna's brother or if the information is outdated). This results in an incorrect answer.
- With reflection: The model successfully (1) evaluates the available content to search for explicit mentions of "childhood memories" and (2) cross-checks past experiences against current artifacts (verifying familial relationships and other reasoning). This enables the system to discard the errors and yield the correct answer: "a handwritten letter."
Q6. Part 1: How is experience aggregated related to an artifact based on different tasks?
Each single-artifact recorded experience includes (i) a task description, which is the task from which this experience entry was derived, and (ii) a contribution summary, which describes how the artifact contributed to solving the task. See Section 3.1. Experience entries are not aggregated per artifact; we store each experience entry.
Q6. Part 2: What happens if over time, the experience-base grows and thus the effort for experience retrieval increases?
Our experience retrieval process is lightweight, ensuring that latency increases slowly as the memory store grows. This process consists of two lightweight steps: a top- vector search over past experiences ( in our experiments), followed by an LLM-based relevance analysis of those top- experiences. For the top- vector search, widely adopted algorithms such as Hierarchical Navigable Small World (https://ieeexplore.ieee.org/abstract/document/8594636) offer a low time complexity of where represents the size of the vector database, while maintaining near-exact empirical accuracy. For the subsequent relevance analysis, because remains a small constant, the processing time stays stable. Consequently, even as stored experiences accumulate over time, the overall latency grows at a slow rate.
Q6. Part 3: What about forgetting or generalizing experience?
These are both additional features that can be added to the ExperienceIndex framework. For example, older experiences can be downweighted and offline analysis passes could aggregate past experiences into more general experiences about an artifact. We leave the exploration of these directions to future work.
Author response to Reviewer GWj7 (Part 1)
Thank you for the thoughtful review and feedback. We address your concerns and answer your questions below.
Thank you for bringing this line of work to our attention and we apologize for missing it. We have revised our paper to discuss ExperienceIndex’s relation to CBR in the related work (see the blue text in Section 2). We reproduce the text below.
Case-based reasoning. Case-based reasoning (CBR) is a longstanding class of techniques that reuse knowledge gained from past problems to help solve new ones (Aamodt & Plaza, 1994; Bach et al., 2025; Wiratunga et al., 2024). ExpIdx shares the underlying principle of the CBR cycle, accumulating and reusing knowledge from past problem-solving experiences. Our work focuses specifically on deriving and storing experiences about artifacts and their relationships, rather than organizing experiences around past problem cases. This artifact-grounded representation helps with experience reuse across different kinds of tasks sharing the same corpus, for example, transferring experiences learned from text-to-SQL (e.g., about table relationships) to factoid question answering over the same tables.
In our evaluation, we ensured that there was no task overlap between the experience tasks (used to construct the memory store) and the evaluation tasks (used to evaluate the agent). In other words, the same task was not present in both task sets.
Figure 4 in Appendix C.1 shows an example where artifact experience improves task solving quality. Here, the agent leverages artifact-pair experience to recognize that the
Order_Items_d9a3table can be joined withProducts_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent did not know to considerProducts_eafband ultimately arrived at the wrong answer.Appendix C shows case studies where each type of experience was used to help an agent solve a task correctly that it was unable to do without the experience.
In Figure 3, the agent leverages single-artifact experience about
session13.mdon a related previous task (“How long has Jolene been doing yoga?”) to correctly answer the task at hand “Which year did Jolene start practicing yoga?” Without this single-artifact experience, the agent was unable to find the appropriate artifact that contained the answer.In Figure 4, the agent leverages artifact-pair experience to recognize that the
Order_Items_d9a3table can be joined withProducts_eafb. It then explores this connection to answer the task at hand “How many distinct orders [...]”. Without this artifact-pair experience, the agent overlookedProducts_eafband ultimately arrived at the wrong answer.In a revision, we will run a full ablation to study how each experience type contributes to the task solving accuracy and cost.
We compute the cosine similarity of the query embedding with respect to the past experience’s task description embedding and contribution summary embedding. We then combine them into one similarity measure by taking their mean.
We used k=5. Yes, k is the upper bound on the number of experiences injected per artifact.
In our memory store database, they are represented as 3-element tuples:
(artifact_id_1, artifact_id_2, relationship)whereartifact_id_1andartifact_id_2are the IDs of the artifacts in the pair andrelationshipis a short natural language description of how the two artifacts are related.