acceptodds
Under review as a conference paper at ICLR 2027

KITE: Retrieval as a Decision Procedure for Long-Term Conversational Memory

Abstract

Many long-term memory systems for LLM agents are built around embedding-based retrieval. Although effective for semantic matching, similarity ranking alone does not establish whether retrieved memories satisfy the conditions implied by a question. In this paper, we introduce KITE (Knowledge-Indexed Temporal Evidence), a vector-free framework that formulates memory retrieval as an explicit decision procedure. KITE organizes multi-session conversations as structured knowledge indexed by a governed topic vocabulary, which evolves through controlled expansion and consolidation. At query time, questions are compiled into inspectable retrieval plans over the same knowledge index that guides memory construction. The plans are then executed through fixed lookup and ranking operations, while sparse lexical retrieval recovers surface details omitted from the structured representation. Experiments on LoCoMo and LongMemEval-S show that KITE achieves 93.5% and 86.0% accuracy while using only about 1.6k reader-context tokens per question. These results demonstrate that conversational memory can move beyond similarity ranking by unifying memory construction and query compilation around a governed index, turning retrieval into an inspectable process for determining which evidence satisfies the question.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-term memory, conversational memory, LLM agents, retrieval-augmented generation, structured retrieval, query compilation, vector-free retrieval, symbolic memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V6PFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KITE, a vector-free framework for long-term conversational memory. It organizes conversations into structured memories using a governed topic vocabulary, then compiles questions into executable retrieval plans with deterministic lookup and ranking operations. A sparse lexical retrieval complements structured retrieval. Experiments on LoCoMo and LongMemEval-S surpass chosen baselines.

Critical strengths
  1. Explicit query planning and structured lookup are useful for long-term memory retrieval.
  2. The method achieves good benchmark results.
Critical weaknesses
  1. The methodological novelty is not sufficiently convincing. The framework mainly combines established techniques, including structured memory extraction, vocabulary management, executable query planning, deterministic lookup, and lexical retrieval.
  2. The contribution of vector-free retrieval is insufficiently distinguished from prior work. For example, Beyond Semantic Similarity [1] has already demonstrated effective retrieval without embedding models or vector indexes through direct corpus interaction.

[1] Li, Zhuofeng, et al. "Beyond semantic similarity: Rethinking retrieval for agentic search via direct corpus interaction." arXiv preprint arXiv:2605.05242 (2026).

Questions for authors
  1. What are the end-to-end token consumption, including memory extraction, vocabulary management, and query compilation? How do these compare with existing systems under consistent evaluation settings?
3 comments
AuthorsOpenReview ↗

Response to V6PF: novelty, vector-free retrieval, and token cost

Thank you for the review. We appreciate the opportunity to clarify the contribution and evaluation scope.

W1: A combination of established techniques?

KITE makes the governed index a shared interface between memory construction and evidence selection (Sections 3.1-3.3). Vocabulary admission and consolidation maintain reusable topic keys. These keys and typed facets organize source-linked facts and ground conditions in executable retrieval plans. The executor checks indexed and temporal conditions before ranking, with explicit relaxation and no-match behavior. The contribution concerns this connection between vocabulary governance, query compilation, and evidence selection, building on established techniques such as structured extraction, vocabulary management, and provenance.

The experiments examine this design at three levels. Table 4 holds the memory store, reader, and reader-context budget fixed: KITE achieves 93.5% LoCoMo accuracy, compared with 85.4% for dense retrieval and 87.8% for hybrid retrieval. Table 5 varies vocabulary-governance policies: fixed vocabulary, immediate expansion, delayed expansion, and full KITE achieve 91.0%, 87.9%, 91.7%, and 93.5%.

In an additional ablation, removing topic/entity labels together with their vocabularies and indexes, while retaining fact text, query-routing logic, and the remaining retrieval rules, reduces accuracy from 93.5% to 88.76%. This measures the contribution of the indexing layer as a whole. Together, these results support the roles of the index, its maintenance, and retrieval over that index.

W2: Beyond Semantic Similarity (DCI)

We agree that being vector-free is not, by itself, a contribution; sparse retrieval such as BM25 has always worked without embeddings. In DCI, an agent searches the raw corpus with terminal tools (grep/rg pipelines, file reads, short scripts). It builds no offline index, enforces exact lexical constraints, and refines its search over many steps. KITE shares the exact matching, and its plans even carry a lexical condition. The differences:

  • Where structure is built. KITE builds its index once, when memory is written, and reuses it for every question. Relative times are resolved to dates at that point. DCI works out its view of the corpus anew for each query.
  • What a condition expresses. KITE's conditions are typed (participant, entity, topic, date range). A fixed executor checks them over indexed fields and can return no match. DCI's conditions are string patterns over raw text, applied by a procedure the agent writes itself.
  • Cost per question. KITE makes about four LLM calls per LoCoMo question. DCI's agents take roughly 20 to 40 tool calls per query in its main settings.
  • DCI was evaluated on document corpora (BrowseComp-Plus, Wikipedia QA, BRIGHT/BEIR), not on conversational memory. We have not compared the two directly, so we do not claim that either is better.

Furthermore, on timing, DCI was posted on arXiv in May 2026 and will appear at NeurIPS 2026 in December, under the ICLR 2027 reviewer guidelines it counts as contemporaneous work. It is clearly relevant, though. We discuss its relationship to KITE in this response.

Q1: End-to-end tokens

For the complete stage-by-stage token and LLM-call breakdown, please see the table under “W3, Q5: Total token cost” in our response to UTnP (2/2). On LoCoMo, KITE uses 8.3k tokens per question: 0.6k for construction, 5.8k for query compilation and search, and 1.9k for answering. EverMemOS uses 12.8k: 6.1k, 2.9k and 3.8k. Both count input plus output tokens. Construction includes extraction, consolidation and vocabulary maintenance, and is divided over the same 1,540 questions.

The settings are only partly consistent. We used the totals EverMemOS published rather than rerunning it, and its counts leave out its 4B embedding and reranking models. EverMemOS is the only compared system that publishes a stage-level breakdown with the same backbone, which is why we compare against it. We have not done this accounting on LongMemEval-S, where construction is not amortized.

Thank you for the clarifications and additional experiments. My reference to DCI was not intended to request a direct comparison, but to emphasize that vector-free retrieval is not itself novel. KITE's contribution therefore lies mainly in its indexing and vocabulary-governance mechanisms, which I find largely engineering-oriented but not methodologically novel. Thus, I would like to maintain my rating.

AuthorsOpenReview ↗

KITE’s contributions to conversational memory retrieval

Thank you for clarifying your point. Our methodological contribution is to make it possible to trace why a particular memory was selected as evidence for a question. Three aspects of KITE work together to make this possible.

  • Retrieval as an explicit decision procedure. KITE checks a question’s conditions against stored memories before ranking them. For a fixed index, validated plan, and execution policy, the same records qualify every time. If the initial conditions yield no matches, the executor applies predefined relaxations. If no record qualifies after those steps, the structured path returns an explicit no-match.

  • A governed index that evolves across conversations. KITE admits new topics as they recur across sessions and consolidates redundant ones, redirecting their entries to a common key. This keeps earlier and later mentions of a topic accessible through the same key, while typed fields preserve distinctions such as who participated in an event and when it occurred.

  • A shared index for writing memories and interpreting questions. The query compiler resolves references in a question against the same topic and entity keys used to index the stored facts. Because each fact links back to its source dialogue, we can trace a match from the wording of the question, through the conditions the compiler inferred, to the utterance that supplied the evidence.

We evaluated plan-guided retrieval on LoCoMo by holding the stored memories, reader settings, and reader-context token budget fixed (Table 4). On those same memories, KITE achieved 87.4% evidence recall compared with 79.9% for hybrid retrieval. Answer accuracy was 93.5% and 87.8%, respectively.

Reviewer_UTnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates how long-term conversational memory should be retrieved: embedding similarity ranks memories without checking whether they satisfy a question's conditions — the right person, event, and time — and fixed top-k retrieval returns candidates even when nothing is relevant. To address it, the authors proposed KITE, which replaces dense retrieval with an explicit decision procedure: sessions are converted into source-linked structured entries under a governed topic vocabulary with typed facets, and each question is compiled into a typed retrieval plan, validated and executed through fixed set operations, deterministic relaxation, and a fixed scoring policy. Different from existing work that focuses on similarity ranking, KITE checks candidate memories against the question's constraints and can return no match. Experimental results verify its effectiveness and superiority: KITE achieves the best overall accuracy on LoCoMo (93.5%) and LongMemEval-S (86.0%), leads temporal reasoning by 8.3 points over the next-best system, and feeds the reader only about 1.5–1.6k context tokens per question.

Critical strengths
  • Large gains on temporal reasoning that match the design. KITE scores 92.2% on LoCoMo temporal questions and 85.7% on LongMemEval-S temporal reasoning, 8.3 points above the next-best system — plausible, since event times are resolved at construction and compiled into explicit time constraints at query time.
  • A genuinely controlled retrieval ablation. With the memory store frozen, plan-guided retrieval beats dense, BM25, and hybrid retrieval on accuracy and recall on both benchmarks (LoCoMo recall 87.4% vs. the hybrid's 79.9%), isolating the retrieval procedure's contribution from memory construction (Table 3).
Critical weaknesses
  • The system is not the deterministic procedure the method section formalizes. The appendix reveals that query compilation draws three candidate plans (one at temperature 0, two at temperature 0.8) and that the reader can trigger an additional search when it judges an answer weakly supported (Appendix B.3). Neither the plan-selection rule nor this follow-up loop appears in the method description, neither is ablated, and no run-to-run variance is reported — so "deterministic and inspectable" holds only for a middle slice of the pipeline.
  • One model does everything, including judging. gpt-4.1-mini performs memory extraction, consolidation, query compilation, and answer generation, and also serves as the LLM judge for the headline accuracy. There is no second judge, human spot-check, or judge-agreement analysis, and whether the copied baseline numbers were produced under an identical judging protocol is not established.
  • The token-efficiency claim counts only the reader's context. The appendix concedes the token metric "measures only the reader-context size rather than the total system token consumption" (Appendix B.1). KITE additionally pays per-session extraction and consolidation calls, three plan compilations per question, and a possible second search — none of it quantified, with no cost or latency comparison against any baseline.
  • Per-category losses to much simpler systems go undiscussed. On LongMemEval-S, KITE trails Naive RAG by 12.5 points on single-session-assistant questions (85.7% vs. 98.2%) and MemOS by over 30 points on single-session-preference (63.3% vs. 96.7%) (Table 2). The text acknowledges only the preference weakness, and never connects the assistant-question gap to its own note that LongMemEval-S extraction "prioritizes the user's statements."
Questions for authors
  • Why was LoCoMo's adversarial category excluded, given that the explicit no-match return is a headline mechanism? What is KITE's abstention accuracy on those questions — and does the system actually abstain, or does the lexical fallback usually supply an answer anyway?
  • With three candidate plans sampled per question, how is the executed plan selected, and what is the end-to-end accuracy variance across repeated runs? In what sense is the system deterministic once plan sampling is included?
  • How often does the reader-triggered follow-up search fire, and what is KITE's accuracy with it disabled?
  • Is the single-session-assistant gap (85.7% vs. Naive RAG's 98.2%) caused by the extraction prompt prioritizing the user's statements — i.e., assistant-stated facts being dropped or deprioritized at construction time?
  • What is KITE's total token cost — construction plus per-question compilation and any follow-up search — compared with EverMemOS end-to-end? The efficiency claim currently rests only on reader-context tokens.
2 comments
AuthorsOpenReview ↗

Response to UTnP (1/2): determinism, plan selection, follow-up, and judging

Thank you for reading the appendix this closely. We clarify these points and provide additional analyses below. This comment covers W1, W2, Q2 and Q3; the next one covers W3, W4, Q1, Q4 and Q5.

W1, Q2: Is the system deterministic?

Determinism is defined at the plan-execution stage (Eq. 2): with the index, the validated plan and the execution policy fixed, the same evidence comes back. Plan generation and answering are LLM calls and do vary between runs. We have added the candidate-selection procedure to Section 3.2 and clarified reader-triggered follow-up in Section 3.3. Appendix B.3 gives the sampling, probing, scoring, and tie-breaking settings. Determinism is defined for a fixed index, validated plan, and execution policy.

Plan selection works as follows. The compiler samples three plans, one at temperature 0 and two at 0.8, and drops malformed ones. Each remaining plan is run on the index (at most 12 rows, 4 per session) and given a fixed heuristic score. The score rises with the number of rows returned and with the use of indexed conditions (topic, entity, entry type), and a plan that returns nothing is penalized. For counting questions, it rewards explicit time bounds instead. A plan that fails during execution is never chosen, and ties go to the earlier plan, so the temperature-0 plan wins ties. Only the selected plan's evidence reaches the reader. If no plan is valid, an empty plan is used and returns no match. We come back to this under Q1.

Run-to-run variation (Q2, Q3)

ConfigurationRun 1Run 2Run 3Mean ± SD
Full KITE93.593.193.393.30 ± 0.20
Single temperature-0 plan93.293.093.193.10 ± 0.10
Without follow-up93.493.093.393.23 ± 0.21

We use multiple candidates to provide alternatives when a generated plan is inadequate, and follow-up to search for evidence missing from the initial retrieval. Values are LoCoMo accuracy (%); SD is the sample standard deviation. The runs without follow-up still sample three plans. Follow-up fired on about 1.8% of questions. Using only the temperature-0 plan lowers the mean by 0.2 points, and removing follow-up lowers it by 0.07. Both are within the run-to-run spread, so on LoCoMo neither mechanism adds much accuracy, while the single-plan setting saves two of the three compilation calls per question. The spread also matters for Table 1. The three-run mean is 93.3%, so on LoCoMo overall, KITE is still better than EverMemOS.

W2: Judging

We regraded the same answers to all 1,540 LoCoMo questions with two more judges. Questions, reference answers, KITE's answers and the judge prompt stayed fixed:

JudgeAccuracy (%)
GPT-4.1-mini (paper)93.5
GPT-4o93.4
GPT-4o-mini95.6

GPT-4o gives almost the same score, and GPT-4o-mini is more lenient. Three annotators also graded 100 questions, sampled at random in proportion to question type. They agreed with the GPT-4.1-mini judge on 98. On the other two, all three annotators judged the answer correct and the LLM judge did not.

As stated in Appendix B.2, the memory-system baseline results are taken from the results consolidated by EverMemOS. The judging protocols differ, as detailed below. The memory-system numbers come from the EverMemOS paper. It reran the LoCoMo baselines and graded each answer with GPT-4o-mini and two auxiliary judges, averaging the three (following MemOS). Its LongMemEval-S baselines come from the MemOS leaderboard. We use a single GPT-4.1-mini judge. GPT-4o-mini alone gives our answers 95.6%, so our judge does not seem to favor KITE. We ran Naive RAG and Full Context ourselves under our protocol. Table 4 holds the memory store, reader settings, judge and context budget fixed.

AuthorsOpenReview ↗

Response to UTnP (2/2): cost, category gaps, and abstention

W3, Q5: Total token cost

The original efficiency analysis focused on reader-context tokens. Below we count every LLM call per LoCoMo question, input plus output tokens. Construction (extraction, consolidation and vocabulary maintenance) is a one-time cost, which we divide over the 1,540 questions. Query compilation and search include follow-up and fallback calls.

StageKITE callsKITE tokensEverMemOS callsEverMemOS tokens
Memory construction0.180.6k4.586.1k
Query compilation and search3.315.8k1.312.9k
Answer generation1.001.9k1.003.8k
Total4.498.3k6.8912.8k

Tokens are rounded (1k = 1,000). KITE's 1.9k answer tokens are the 1.5k evidence context from Table 1 plus instructions and output. The EverMemOS figures are the GPT-4.1-mini totals in Table 8 (Appendix A.3) of the EverMemOS paper, divided by the same 1,540 questions.

At GPT-4.1-mini list prices ($0.40 / $1.60 per 1M input / output tokens), this is USD 0.0043 vs. 0.0069 per question, or 35% fewer tokens and 38% lower cost. All of the saving comes from construction. At query time KITE uses more tokens (7.7k vs. 6.7k), mostly for its three compiled plans. With these rounded token counts, KITE uses fewer total LLM tokens up to roughly 850 questions per conversation; LoCoMo has about 154. This accounting covers LoCoMo only. On LongMemEval-S each history gets one question, so construction is not amortized.

W4, Q4: Single-session-assistant questions

Partly, yes. The LongMemEval-S extraction prompt keeps mostly what the user says (Appendix C.1). Across the 500 histories, extracted facts cite 85.8% of user turns but only 2.4% of assistant turns. Only 1 of the 51 gold assistant turns behind the 56 assistant questions is cited by any fact. The structured path contributes limited direct evidence for this category. The 85.7% comes from the lexical channel, which searches the raw dialogue.

Table 2 shows different strengths across question types: KITE and Naive RAG score 82.0% versus 48.9% on multi-session questions and 85.7% versus 39.8% on temporal questions. Preserving complete assistant evidence remains a concrete limitation of the current implementation.

Preference questions

Preference questions remain challenging for the current system. One caveat on the size of the gap: preference answers are graded against a rubric, and the score is sensitive to it. The same 30 answers score 63.3% (19/30) with LongMemEval's preference prompt and 90.0% (27/30) when GPT-4o-mini grades them with a generic correctness prompt. That check changes both the judge and the prompt. So it shows the score is sensitive to grading, but not how much of the gap to MemOS comes from grading. We report 63.3%.

Q1: Adversarial questions and abstention

We excluded the adversarial category from our main comparison to follow the scope of prior LoCoMo evaluations. Mem0 explicitly excludes adversarial questions (Chhikara et al., 2025, Section 3.1), while MemOS (Li et al., 2025, Section 6.1, Table 3) and EverMemOS (Hu et al., 2026a, Section 4.1, Table 1) report results for the four non-adversarial categories without reporting adversarial-category scores. It is the natural test of no-match, so we have now run all 446 adversarial questions with the full system (structured path, lexical fallback and follow-up), without telling the reader that the questions are unanswerable. A GPT-4.1-mini judge counted 368/446 (82.5%) as correct abstentions. The GPT-4.1-mini judge ran at temperature 0 using the unchanged official LongMemEval abstention template for all configurations. This is our own open-ended setup, not LoCoMo's option-based evaluation, so the number is not comparable to published adversarial scores.

Does the fallback usually supply an answer anyway? No. The structured path returned no match on 260 questions (58.3%). All of these were genuine zero-match results, not compilation failures. On 247 of them (95.0%), the reader declined to answer. On 13, it answered from fallback candidates, and 9 of those answers repeated the false premise. The other 186 questions matched some structured evidence, and the reader answered far more often there. So the main challenge is not the fallback. It is false-premise questions that still match real evidence, and the selection score's preference for non-empty plans pushes in the same direction. On LongMemEval-S, KITE correctly abstains on 24 of the 30 abstention questions (80.0%). We have added this analysis to Appendix D.6.

Reviewer_DxYxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed and studied a new approach to management of the memory of LLM agents. The proposed approach, called Knowledge-Indexed Temporal Evidence (KITE) is a vector-free framework that frames memory retrieval problem as an explicit decision procedure and attempts to turn raw session information into well-organized structured knowledge, which can then provide evidence for supporting question answering. This allows for retrieving evidence relevant to the answer to a question. As such a structured matching approach may sometimes return no results, the framework is also extended to use lexical retrieval approaches. Evaluation results show that KITE outperforms multiple state-of-the-art memory management approaches. Detailed analysis revealed understanding of strengths and weaknesses of KITE (e.g., its performance on preference questions is not good, which the authors suggest to further investigate in the future).

Critical strengths
  • memory management is an important yet challenging problem in developing intelligent agent. No established solution to the problem yet, so this work makes a useful contribution, though there are many approaches proposed already.
  • The overall idea of KITE is appealing as it attempts to focus on directly retrieving what's needed to answer a question, instead of ranking retrieved results and post-processing the results to generate an answer
  • Experiment results are comprehensive and reveal interesting, deeper understanding of the behavior of the proposed KITE.
  • The paper is written well and easy to follow.
Critical weaknesses
  • A main weakness is that the paper appears to argue against vector-based approaches, in favor of the more interpretable symbolic/structured approaches. I'm not sure whether this argument is solid as dense vector-based retrieval also has advantages. Indeed, one advantage is that it could address the problem of "no results" in the proposed structured retrieval approach. In the paper, this deficiency was addressed by using a lexical retrieval approach, but pure keyword matching may still be insufficient (e.g., synonyms aren't matched); instead, a neural ranking model may be more robust.
  • The related work part provides a clear review of the existing work, but how the proposed KITE is related to each line of related work was not discussed. As a result, it is unclear which part of KITE is really new and which is taken from an existing work. Overall, the novelty of the proposed KITE as compared with many existing approaches is not so clear. There is also some other work that might be even more related to KITE. For example, the following ICML'26 paper also used structured representation with provenance. There should be a discussion of the similarity and difference: PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents, Ke Yang*, Zixi Chen*, Xuan He*, Jize Jiang*, Michel Galley, Chenglong Wang, Jianfeng Gao, Jiawei Han, and ChengXiang Zhai. https://openreview.net/forum?id=NWKaQIKoGp
  • The structured knowledge matching is accurate (if there's a match), but may not be able to handle inexact matching that may be needed; instead an embedding-based representation of the structured knowledge may be more robust, but this wasn't explored or discussed.
Questions for authors
  • Can you clarify the novelty of the techniques used in the proposed KITE in the context of all the previous work, including PlugMem? Which parts are from existing work and which parts are new?
  • Have you considered replacing the lexical retrieval with a dense vector retrieval approach?
  • Have you considered learning embedding representation of entities and relations so that you could use embedding vector matching to retrievel evidence for question answering?
1 comment
AuthorsOpenReview ↗

Response to DxYx: what is new, PlugMem, and dense retrieval

Thank you for your thoughtful review and for recognizing the value of KITE's retrieval approach, the depth of our experimental analysis, and the clarity of the paper. We also appreciate your suggestions on PlugMem and dense retrieval.

Q1, W2: What is borrowed and what is new

Some components come from prior work:

  • extracting structured facts from dialogue: Mem0, A-MEM, PlugMem
  • canonicalizing labels to limit fragmentation: EDC, KGGen
  • executable operations generated from natural-language requests: ChatDB, Text2Mem, KAG
  • time-aware memory: Zep, TReMu
  • posting-list evaluation, step-wise relaxation, BM25 and rank fusion: standard IR

Our major contribution is how vocabulary governance and query compilation are tied together. One governed vocabulary is both the target of extraction and the language of retrieval plans. A topic is admitted only after it recurs in three sessions across at least two months, and redundant siblings are merged. The compiler must write a plan's conditions in exactly those keys, and the executor checks them before ranking, with explicit relaxation and no-match.

Table 5 changes only the governance policy. LoCoMo accuracy moves from 87.9% (immediate expansion) and 91.0% (fixed) to 91.7% (delayed) and 93.5% (full KITE). In an additional ablation, we removed the topic and entity labels together with their vocabularies and indexes, and kept the fact text, query routing and remaining retrieval rules. Accuracy drops from 93.5% to 88.8%. We have made this mapping explicit in Section 2.

PlugMem

PlugMem is close to KITE in several ways. Both extract atomic facts linked to their source, attach concept or topic tags that act as an index, use an LLM to turn the question into retrieval signals, and merge near-duplicate memories. From the arXiv version (Sections 3.1-3.2, Appendices C.2-C.3), we see three main differences:

  • Vocabulary. PlugMem's concept tags are open-ended, mostly copied verbatim from each fact, and type labels are explicitly disallowed. KITE admits topics into a governed vocabulary and adds typed facets.
  • Plans. PlugMem's plan is free text (tags and a subgoal) that routes the search. KITE's plan is a typed structure validated against the index.
  • Execution. In PlugMem, tag matches add candidates, which are merged with dense candidates and reranked over LLM-controlled hops. Nothing is excluded before ranking, and there is no no-match outcome. On KITE's structured path, a memory must satisfy the plan's conditions, including time ranges, before it is ranked.

The distinction lies in how the shared vocabulary and executable conditions determine evidence selection, rather than in having structure or provenance alone. The plans and source links also make the applied conditions and supporting dialogue inspectable. This provides a basis for diagnosing errors. Improved debugging efficiency is a potential benefit for future evaluation.

W1, Q2: Dense retrieval

We have not yet evaluated dense retrieval as a replacement for the lexical channel. We chose BM25 because the channel's job is to recover surface details that the facts leave out, such as rare names and numbers (Section 3.3), and exact matching suits that. Your point still stands, though. In Table 4, dense retrieval beats BM25 as a standalone retriever on our memory store (85.4% vs. 81.3% on LoCoMo). On LongMemEval-S assistant questions, where KITE depends on the lexical channel, dense Naive RAG reaches 98.2%. A neural reranker also helps our multi-hop questions (95.0% vs. 93.3%, Table 6). So a dense or hybrid supplementary channel may well help, and we see it as the most direct next experiment. We have revised the introduction to focus on explicit condition evaluation, while recognizing the value of semantic matching. Appendix D.4 discusses dense or hybrid supplementary retrieval as a direction for future evaluation.

W3, Q3: Entity and relation embeddings

We have not trained entity or relation embeddings in the present study. Inexact matching currently happens at compile time. The LLM compiler reads the question together with the stored vocabulary and entity list and maps it to existing keys, so the question need not repeat the stored wording. The executor then works only with keys. If the compiler picks the wrong key, or no key fits, the structured path misses the evidence. Our lowest-scoring category, preference questions, is partly this case. Embeddings could help in two places: suggesting the nearest existing key when a reference cannot be resolved at validation, and shortlisting keys for the compile prompt. This is a compatible extension that requires separate evaluation. We interpret your suggestion as learning embeddings of the structured entities and relations to support inexact matching. Please let us know if you had a different formulation in mind.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.