acceptodds
Under review as a conference paper at ICLR 2027

LEDGER: Memory as Computation for Conversational Agents

Abstract

Long-term conversational memory is often treated as a retrieval problem, yet many questions require computation over evidence distributed across interactions. Retrieving relevant memories does not specify which facts to count, combine, or compare. We introduce Long-term Evidence Decomposition for Grounded Executable Reasoning (LEDGER), which makes memory aggregation an explicit computation rather than an implicit step in answer generation. LEDGER extracts reusable, source-linked facts independently of future queries. A semantic router uses candidate facts to specify what to compute, while a deterministic executor evaluates the resulting restricted program over the full fact store. Valid computed records supplement rather than replace retrieved raw evidence. On LongMemEval, LEDGER improves accuracy from 64.4% to 68.9%, with gains from 70.1% to 76.6% on multi-session questions and from 65.1% to 82.5% on knowledge updates. On LoCoMo, accuracy improves from 57.3% to 61.8% against the same baseline. These results support treating conversational memory not merely as context to retrieve, but as evidence to compute over.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-term conversational memory, agent memory, LLM agents, retrieval-augmented generation, semantic parsing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BB3aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LEDGER separates fact extraction, program induction, execution, and answering for conversational-memory aggregation tasks: counting, summation, differences, ratios, latest-state selection, and listing. An offline extractor produces source-linked facts session by session without seeing future queries. A Phi-3 router fine-tuned on synthetic MSC data uses candidate facts to specify an operator, filters, and operand bindings. A deterministic executor evaluates the program over the complete fact store and appends the result, supporting fact IDs, and source IDs to Parent RAG’s original top-10 sessions. NONE, empty support, or invalid results return the original retrieval payload.

Critical strengths
  1. Fact extraction is independent of future queries. The router explicitly specifies operations and filters, and the executor returns results with supporting sources. Candidate selection bounds the program-induction context while execution covers the complete fact store. This separation supports auditing omissions, binding errors, and execution errors independently and reusing the same facts for subsequent queries.

  2. LEDGER and Parent RAG share the raw retrieval policy, with gains of 4.5 percentage points on both benchmarks. In Table 8, replacing the executor with an LLM reduced test402 accuracy to 66.2%, and removing the raw-text overlay reduced it to 64.2%. These comparisons support combining execution with raw evidence. The paper also distinguishes earlier privileged diagnostics from final results.

Critical weaknesses
  1. Section 6’s 11/13 operator accuracy and 4/13 execution equivalence concern early SFT, not V11.3. The final system lacks measurements of non-NONE coverage, support-set accuracy, operand binding, and execution equivalence. The results therefore do not identify which computations produce the end-to-end gains or establish whether the final router resolves the binding bottleneck identified in the paper.

  2. Table 2 compares retrieval variants, PAL-style, and Fact Store + LLM, but no structured-memory method. PAL also changes both fact-construction timing and input context. Table 8 supports the executor’s value within this system, but does not separate the contributions of reusable extraction, router training, and the restricted language relative to related alternatives.

  3. The final payload does not add program-support sessions. If supporting sources fall outside Parent’s top-10, the answer model cannot necessarily check the side channel against raw text. Structural validation cannot detect semantic errors, and the reported configuration disables confidence gating. The overlay ablation supports retaining raw text but does not measure how often incorrect computations are accepted, corrected, or cause regressions.

  4. LEDGER adds per-session fact extraction, router training, and per-query program induction while retaining ten complete raw sessions. Construction cost, fact-store size, query tokens, latency, and amortized cost across query counts are unreported. Without these measurements, the practical value and operating range of reuse relative to query-time PAL or Fact Store + LLM remain unresolved.

Questions for authors
  1. What are V11.3’s non-NONE coverage, support-set precision/recall, operand-binding accuracy, and execution equivalence by operator? How often do supporting sources fall outside Parent’s top-10? For structurally valid but semantically incorrect records, please trace their acceptance or correction rates and the resulting answer changes relative to Parent.

  2. With identical facts, raw-text payloads, and training supervision, how do prompted program induction and the fine-tuned router compare? Could you add a structured-memory baseline with the same answering model and quantify the separate gains from the fixed fact store and restricted execution interface relative to existing combinations?

  3. Please report per-question improvements/regressions and paired intervals against Parent RAG and the executor ablation, resampling LoCoMo by conversation. Among the remaining 232 questions, are regressions offset by improvements, and in which question types? Do overall and target-type differences persist across repeated runs or evaluation by an independent judge?

  4. Please report extraction/training cost, routing and answering tokens per query, fact-store storage, and latency, with amortization across queries over the same history. Against query-time PAL and Fact Store + LLM, what total costs are required to achieve the reported accuracy, and at what query counts do the costs cross over?

Reviewer_pgxMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that many long-term conversational memory questions require aggregation over facts across sessions and just doing retrieval is not sufficient. It proposes LEDGER, which extracts typed facts from each session offline, uses a fine-tuned semantic router to map a query into one of six aggregation operators (count, sum, diff, divide, latest, list) with filters, executes it deterministically over the fact store, and appends the result. On LongMemEval-S, accuracy improves while concentrated on multi-session and knowledge-update questions. Ablation studies and dev-set diagnostics are provided as additional analysis.

Critical strengths
  • The motivation that retrieval hit does not imply correct integration is sound, and the coverage study supports it as a good motivation
  • The ablation studies and error analysis provides deeper understanding of the mechanism
Critical weaknesses
  • The "programs" are six hand-curated operators over numerical/entity values, closer to query templates than program induction. Plus the operator set appears to be fit to the LongMemEval question types. The title and framing oversell what is done.
  • All baselines are the RAG variants; no published memory system is run which from my perspective is necessary for this work. A large part of Sec. 5 and App. B/D is diagnostics on the dev set, each explicitly labeled non-controlled, development-specialized or privileged These cannot support claims and take space that per-type test results and external baselines should have.
  • Offline fact extraction is one LLM call per session, the router is an extra model call per query, and the fact store grows with history, but no token/latency/cost numbers are given for building the store or for inference (admitted L461).
  • Missing details and presentation. The router model appears only in Appx, not in the main text. "V11.3" and "Semantic Router v2" seem to internal version names never defined.
  • All LLM components (extraction, answer, judge) use Claude Haiku 4.5 only.
Questions for authors
  • Can the authors also report per-question-type accuracy on test402 for comparison, and the fraction of test queries routed to a non-NONE program.
  • What is the cost (tokens, calls, latency) of building the fact store and of inference compared to Parent RAG?
Reviewer_bw89OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors claim to introduce LEDGER, to make memory aggregation an explicit computation, where it extracts reusable facts independently of future queries. A router specifies what to compute based on the candidate facts that they have retrieved. LEDGER shows higher performance of accuracy over prior work (ParentRAG) on LongMemEval and LoCoMo of 4.5pp for both. They claim that conversational memory should not be handled only as a context to retrieve, but as evidence that computation should be done on.

Critical strengths
  • The design is simple, and the ablation shows that keeping the raw text is important.
Critical weaknesses
  • The paper writing and structure make the whole paper hard to interpret and follow. From unclear writing with unclear claims, it is hard to follow what the authors are actually trying to claim. For this to be a publishable piece, a huge amount of rewriting should be done along with concrete experiments and comparisons denoted below.

  • Experiments are insufficient. For instance, none of the memory systems such as Mem-0 are compared. A 18 question probing cannot support a claim to be made, a larger scale is a must. Variation in model sizes and families seem needed. For the technique itself, each of the components (as the authors claim) are existing in other systems such as Mem-0 or A-MEM, with this paper integrating these different techniques (facts are extracted in advance, the router only writes a program, and the computed result is added to the retrieved text). One might claim the novelty of this combination itself, but the integration done by the authors in this paper is too limited (only 6 operations) and is way too simple without careful modifications to each technical component for better integration. Moreover, the most challenging part is not addressed (choosing the right facts and putting them in the right operands).

Questions for authors
  • Why are there unnecessary version numbers such as Ledger V11.3? This seems like a development code number.
  • Why is appendix J missing?
  • What are the costs of the system relative to other RAG methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.