- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
INCUMBENT proposes a structured memory system for conversational agents, targeting loss of original details, confusion between facts and inferences, and stale-value retrieval. An LLM proposes updates, while a deterministic store checks source-span occurrence, assigns provenance labels from vertex types, and maintains current and historical values per attribute key. The reader prioritizes active records and retrieves original dialogue text with turn/status labels. Experiments against six baselines report 95.0% truthfulness on AI-LieDar, a 0.344 memory score on AMemGym, and 59.0 mean F1 on LoCoMo. A crossed ablation further shows that recording superseded values helps substantially only when the reader respects their status.
Critical strengths
-
The separation between model-proposed updates and rule-based memory operations is useful. Explicit current/history records and status-aware reading address stale-value retrieval, while source-span retrieval preserves details that can be lost in summaries.
-
The results are strong across three benchmarks, with repeated runs and analyses at the user, scenario, or conversation level. The crossed record-by-reader ablation is particularly informative: recording superseded values helps substantially only when the reader respects those records.
Critical weaknesses
-
The span check only confirms that the text appears in the conversation; it does not check whether the text supports the stored claim. The LLM still decides whether an item is a fact or an inference, and propositions are kept even when their spans fail validation. The paper does not show how often these errors occur or how much they affect answers.
-
The cost comparison leaves out actual maintenance-call totals, output/reasoning tokens, and the full memory-construction cost on LoCoMo. With memory updates every two turns and longer wall-clock times than several baselines, the low-cost claim needs more evidence. The maintainer also reads full lineages that can keep growing even when the attribute count stays fixed, so Equation 4 does not justify the claimed write-cost scaling.
Questions for authors
-
Can the authors check a representative sample for unsupported propositions and fact/inference classification errors, and report their impact on answers?
-
Can the authors report maintenance calls, input/output tokens, API cost, and memory-construction time per LoCoMo conversation for each method? How does performance change with less frequent updates?
-
What limits the length of the lineages shown to the maintainer? How is the claimed write-cost scaling justified when these histories keep growing?
3 comments
[W2 / Q2] Full LoCoMo memory-construction costs and update-cadence ablation
Below we report the per-conversation LoCoMo memory-construction costs across all agents (mean over the 10 conversations), measured under the exact same accounting as Table 11 (all direct API and embedding calls, excluding cached fixed instruction blocks). API cost is computed at standard gpt-5.4-mini pricing ($0.75 / 1M input tokens, $4.50 / 1M output tokens) and text-embedding-3-small pricing ($0.02 / 1M tokens):
| Agent | Build Calls | Build Input (M) | Build Output (M) | Build API Cost | Build Time (min) |
|---|---|---|---|---|---|
| Pure LLM | 0 | 0 | 0 | $0 | 0 |
| RAG | 0 (588 embed) | 0.018 | 0 | $0.0004 | 4 |
| A-MEM | 1,176 | 1.10 | 0.65 | $3.75 | 140 |
| AWI | 294 | 0.25 | 0.015 | $0.26 | 5 |
| Mem0-G | 1,460 | 0.58 | 0.20 | $1.34 | 310 |
| AWE | 566 | 0.18 | 0.10 | $0.59 | 35 |
| INCUMBENT | 294 | 1.08 | 0.12 | $1.35 | 30 |
Three points stand out from this table:
- Fewest maintenance calls. Because INCUMBENT updates once every turns, it makes 294 build calls per conversation—tied with AWI for the fewest among all LLM-maintained memories, roughly half of AWE (566), and one-fourth to one-fifth of A-MEM (1,176) and Mem0-G (1,460).
- Wall-clock construction time. INCUMBENT builds a LoCoMo conversation's memory in 30 minutes, on par with AWE (35 min), and roughly faster than A-MEM (140 min) and faster than Mem0-G (310 min). This ordering matches AI-LieDar in Table 11 (whereas AMemGym s/turn in Table 11 includes generating long per-turn assistant replies alongside maintenance).
- Total cost including QA. Combining construction and question answering across a conversation, INCUMBENT uses 1.39M total input tokens against 2.82M for Pure LLM (which reads the full transcript on every question), or $1.58 vs. $2.11 per conversation in total API cost. Construction cost is amortised after 66 questions in input tokens and 110 questions in dollar cost.
At the same time, your point is well taken: INCUMBENT's standalone construction cost on LoCoMo is not small. Its build input (1.08M tokens) exceeds AWE (0.18M), AWI (0.25M), and Mem0-G (0.58M), and its build API cost (\$1.35) is higher than AWI (\$0.26) and AWE (\$0.59), equal to Mem0-G (\$1.34), and lower than A-MEM (\$3.75). This is because the maintainer reads the current value of every attribute on each call (Equation 3): LoCoMo utterances are short, making per-turn extraction prompts small for other agents, whereas INCUMBENT's write input scales with store size. However, those cheaper builders score 10.0 to 17.0 F1 points below INCUMBENT on LoCoMo (Table 1).
We therefore explicitly scope our efficiency claims in the paper to three statements:
- On AMemGym, INCUMBENT uses the fewest total input tokens per user turn of any agent (
5.4Kvs.8.0–25.0K, Table 11). - On LoCoMo, its reader context per question is smaller than a transcript agent (
2,000vs.18,307tokens, Table 11). - Maintenance input per call grows with the size of the store rather than the length of the transcript (Figure 4).
Finally, regarding less frequent updates: relaxing the cadence from every 2 turns to every 4 turns () halves LoCoMo construction calls to 147 and build input tokens to 0.55M, while Mean F1 moves only slightly from 59.0 to 57.8.
[Q3] Lineage length and the write-cost bound in Equation 4
Your mathematical point is valid. While the number of lineages shown to the maintainer is bounded by , the length of an individual lineage has no structural upper bound, so does not follow from Equation 4 alone. We therefore revise the stated bound in Proposition 5 to . Here is the number of state changes on a single attribute, which remains small in practice: empirically, mean lineage length is 2.4 on LoCoMo (max 8) and 3.1 on AMemGym (max 11), and lineages account for only 5% (LoCoMo) and 13% (AMemGym) of the input tokens on the final maintainer call. Thus, in practice, write input is still dominated by the attribute count : as conversation length grows on LoCoMo, maintainer input grows by (Figure 4).
We hope these measurements and clarifications fully address your concerns, and we welcome any additional questions.
Thank you for the additional analyses. I appreciate that the rebuttal addresses my questions, but I feel that some of the new results narrow rather than strengthen the broader claims. As acknowledged by the authors, the LoCoMo results show that construction is not particularly low-cost, while the original scaling claim in Eq. 4 also requires correction. The empirical lineage statistics are useful, but do not provide a general scalability guarantee. I am currently inclined to maintain my score of 2 while awaiting other reviewers' responses.
Thank you for your constructive review and for requesting concrete empirical measurements of error rates and memory-construction costs. We have carried out each of the requested analyses and report them below.
[W1 / Q1] Frequency of proposition/classification errors and their impact on answers
As you pointed out, exact substring containment verifies that a span occurred in the dialogue, while the model still decides whether the span supports the proposition and whether an item is a fact or an inference. To measure how often these errors occur, we drew a stratified random sample of 900 vertices across the three benchmarks (495 facts, 225 concerns, 180 hypotheses). Each vertex was evaluated by a separate judge model (
Claude Sonnet 5.5) and verified by the authors: we manually inspected all 42 flagged errors (confirming 38) and a random sample of 150 unflagged vertices (confirming 149/150 agreement).facthypothesisTo measure the downstream impact of these errors on question answering, we applied the same evaluation to the gold-attribute vertices of all 2,200 AMemGym items. An error was present on the relevant attribute in 185 items (8.4%); accuracy on those 185 items is
0.300, compared with0.480on the remaining 2,015 items. Consequently, even if every such error were eliminated and those items were answered at the0.480rate, overall AMemGym accuracy (.465) would increase by at most 1.5 pp. We will incorporate this analysis and explicitly state the limitation in the paper: the span check verifies verbatim occurrence, and a vertex whose span fails validation passes only its proposition to the reader (Appendix A).