acceptodds
Under review as a conference paper at ICLR 2027

INCUMBENT: THE MODEL PROPOSES, THE STORE DISPOSES — STRUCTURAL TRUTH MAINTENANCE FOR AGENT MEMORY

Abstract

Long-horizon agents let a language model write their memory: it paraphrases what the user said, infers what they want, and decides when a fact has expired, so the model’s own output becomes state that is later retrieved as evidence. We introduce INCUMBENT, a hypergraph memory in which the model proposes and the store disposes: every span it keeps is verified against the user’s words, the store assigns provenance so an inference is never read as a user statement, and each attribute holds one current value to which the reader is bound. Against six baselines, INCUMBENT lifts truthfulness on AI-LieDar to 95.0% from the best baseline’s 53.3%, raises AMemGym’s memory score by 67% (0.344 against 0.206), and reaches 59.0 F1 on LoCoMo while reading 9× fewer context tokens than an agent that holds the transcript. Enforcement, not recording, is what pays: a record of superseded values is worth 0.2 percentage points of AMemGym accuracy to a reader that ranks by similarity and 5.7 to one bound by it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon dialogue agents, LLM memory, memory architecture, hypergraph memory, knowledge updating, fact supersession, conversational agents, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a8dvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

INCUMBENT proposes a structured memory system for conversational agents, targeting loss of original details, confusion between facts and inferences, and stale-value retrieval. An LLM proposes updates, while a deterministic store checks source-span occurrence, assigns provenance labels from vertex types, and maintains current and historical values per attribute key. The reader prioritizes active records and retrieves original dialogue text with turn/status labels. Experiments against six baselines report 95.0% truthfulness on AI-LieDar, a 0.344 memory score on AMemGym, and 59.0 mean F1 on LoCoMo. A crossed ablation further shows that recording superseded values helps substantially only when the reader respects their status.

Critical strengths
  1. The separation between model-proposed updates and rule-based memory operations is useful. Explicit current/history records and status-aware reading address stale-value retrieval, while source-span retrieval preserves details that can be lost in summaries.

  2. The results are strong across three benchmarks, with repeated runs and analyses at the user, scenario, or conversation level. The crossed record-by-reader ablation is particularly informative: recording superseded values helps substantially only when the reader respects those records.

Critical weaknesses
  1. The span check only confirms that the text appears in the conversation; it does not check whether the text supports the stored claim. The LLM still decides whether an item is a fact or an inference, and propositions are kept even when their spans fail validation. The paper does not show how often these errors occur or how much they affect answers.

  2. The cost comparison leaves out actual maintenance-call totals, output/reasoning tokens, and the full memory-construction cost on LoCoMo. With memory updates every two turns and longer wall-clock times than several baselines, the low-cost claim needs more evidence. The maintainer also reads full lineages that can keep growing even when the attribute count stays fixed, so Equation 4 does not justify the claimed write-cost scaling.

Questions for authors
  1. Can the authors check a representative sample for unsupported propositions and fact/inference classification errors, and report their impact on answers?

  2. Can the authors report maintenance calls, input/output tokens, API cost, and memory-construction time per LoCoMo conversation for each method? How does performance change with less frequent updates?

  3. What limits the length of the lineages shown to the maintainer? How is the claimed write-cost scaling justified when these histories keep growing?

3 comments

Thank you for your constructive review and for requesting concrete empirical measurements of error rates and memory-construction costs. We have carried out each of the requested analyses and report them below.

[W1 / Q1] Frequency of proposition/classification errors and their impact on answers

As you pointed out, exact substring containment verifies that a span occurred in the dialogue, while the model still decides whether the span supports the proposition and whether an item is a fact or an inference. To measure how often these errors occur, we drew a stratified random sample of 900 vertices across the three benchmarks (495 facts, 225 concerns, 180 hypotheses). Each vertex was evaluated by a separate judge model (Claude Sonnet 5.5) and verified by the authors: we manually inspected all 42 flagged errors (confirming 38) and a random sample of 150 unflagged vertices (confirming 149/150 agreement).

Error TypeRate
Span containment check failed (all vertices, automated count)4.3%
Verified span does not entail/support its proposition (among the 689 facts & concerns passing span check)3.3% (23 / 689)
Model inference misclassified as fact2.0% (10 / 495)
User statement misclassified as hypothesis2.8% (5 / 180)

To measure the downstream impact of these errors on question answering, we applied the same evaluation to the gold-attribute vertices of all 2,200 AMemGym items. An error was present on the relevant attribute in 185 items (8.4%); accuracy on those 185 items is 0.300, compared with 0.480 on the remaining 2,015 items. Consequently, even if every such error were eliminated and those items were answered at the 0.480 rate, overall AMemGym accuracy (.465) would increase by at most 1.5 pp. We will incorporate this analysis and explicitly state the limitation in the paper: the span check verifies verbatim occurrence, and a vertex whose span fails validation passes only its proposition to the reader (Appendix A).

[W2 / Q2] Full LoCoMo memory-construction costs and update-cadence ablation

Below we report the per-conversation LoCoMo memory-construction costs across all agents (mean over the 10 conversations), measured under the exact same accounting as Table 11 (all direct API and embedding calls, excluding cached fixed instruction blocks). API cost is computed at standard gpt-5.4-mini pricing ($0.75 / 1M input tokens, $4.50 / 1M output tokens) and text-embedding-3-small pricing ($0.02 / 1M tokens):

AgentBuild CallsBuild Input (M)Build Output (M)Build API CostBuild Time (min)
Pure LLM000$00
RAG0 (588 embed)0.0180$0.00044
A-MEM1,1761.100.65$3.75140
AWI2940.250.015$0.265
Mem0-G1,4600.580.20$1.34310
AWE5660.180.10$0.5935
INCUMBENT2941.080.12$1.3530

Three points stand out from this table:

  1. Fewest maintenance calls. Because INCUMBENT updates once every κ=2\kappa = 2 turns, it makes 294 build calls per conversation—tied with AWI for the fewest among all LLM-maintained memories, roughly half of AWE (566), and one-fourth to one-fifth of A-MEM (1,176) and Mem0-G (1,460).
  2. Wall-clock construction time. INCUMBENT builds a LoCoMo conversation's memory in 30 minutes, on par with AWE (35 min), and roughly 5×5\times faster than A-MEM (140 min) and 10×10\times faster than Mem0-G (310 min). This ordering matches AI-LieDar in Table 11 (whereas AMemGym s/turn in Table 11 includes generating long per-turn assistant replies alongside maintenance).
  3. Total cost including QA. Combining construction and question answering across a conversation, INCUMBENT uses 1.39M total input tokens against 2.82M for Pure LLM (which reads the full transcript on every question), or $1.58 vs. $2.11 per conversation in total API cost. Construction cost is amortised after 66 questions in input tokens and 110 questions in dollar cost.

At the same time, your point is well taken: INCUMBENT's standalone construction cost on LoCoMo is not small. Its build input (1.08M tokens) exceeds AWE (0.18M), AWI (0.25M), and Mem0-G (0.58M), and its build API cost (\$1.35) is higher than AWI (\$0.26) and AWE (\$0.59), equal to Mem0-G (\$1.34), and lower than A-MEM (\$3.75). This is because the maintainer reads the current value of every attribute on each call (Equation 3): LoCoMo utterances are short, making per-turn extraction prompts small for other agents, whereas INCUMBENT's write input scales with store size. However, those cheaper builders score 10.0 to 17.0 F1 points below INCUMBENT on LoCoMo (Table 1).

We therefore explicitly scope our efficiency claims in the paper to three statements:

  1. On AMemGym, INCUMBENT uses the fewest total input tokens per user turn of any agent (5.4K vs. 8.0–25.0K, Table 11).
  2. On LoCoMo, its reader context per question is 9×9\times smaller than a transcript agent (2,000 vs. 18,307 tokens, Table 11).
  3. Maintenance input per call grows with the size of the store rather than the length of the transcript (Figure 4).

Finally, regarding less frequent updates: relaxing the cadence from every 2 turns to every 4 turns (κ=4\kappa = 4) halves LoCoMo construction calls to 147 and build input tokens to 0.55M, while Mean F1 moves only slightly from 59.0 to 57.8.

[Q3] Lineage length DD and the write-cost bound in Equation 4

Your mathematical point is valid. While the number of lineages shown to the maintainer is bounded by NtN_t, the length DD of an individual lineage has no structural upper bound, so Θ(∣E∣)\Theta(|\mathcal{E}|) does not follow from Equation 4 alone. We therefore revise the stated bound in Proposition 5 to O((∣E∣+∣Cact∣+Nt⋅D)⋅L)O((|\mathcal{E}| + |C_{\text{act}}| + N_t \cdot D) \cdot L). Here DD is the number of state changes on a single attribute, which remains small in practice: empirically, mean lineage length is 2.4 on LoCoMo (max 8) and 3.1 on AMemGym (max 11), and lineages account for only 5% (LoCoMo) and 13% (AMemGym) of the input tokens on the final maintainer call. Thus, in practice, write input is still dominated by the attribute count ∣E∣|\mathcal{E}|: as conversation length grows 28×28\times on LoCoMo, maintainer input grows by 4.2×4.2\times (Figure 4).

We hope these measurements and clarifications fully address your concerns, and we welcome any additional questions.

Thank you for the additional analyses. I appreciate that the rebuttal addresses my questions, but I feel that some of the new results narrow rather than strengthen the broader claims. As acknowledged by the authors, the LoCoMo results show that construction is not particularly low-cost, while the original scaling claim in Eq. 4 also requires correction. The empirical lineage statistics are useful, but do not provide a general scalability guarantee. I am currently inclined to maintain my score of 2 while awaiting other reviewers' responses.

Reviewer_fqrMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

INCUMBENT maintains a per-attribute memory in which a model proposes updates and a deterministic store applies span checks, provenance fields, status transitions, and a one-current-value-per-key rule. At read time, values marked as current are given priority. The paper evaluates the method on AI-LieDar for truthfulness, AMemGym for state tracking, and LoCoMo for conversational QA, with comparisons against six baselines. It also includes several ablations, including a crossed comparison between the supersession record and the reader's use of status information.

Critical strengths
  1. The separation between model-proposed updates and store-enforced state transitions is a useful design choice. In particular, when a new state fact is written to an occupied attribute, the previous current value is superseded (Proposition 2), giving a clear and testable per-key update rule.

  2. Table 2 gives a useful crossed ablation of the supersession record and the reader rule. Keeping the supersession record improves accuracy by 5.7 pp when the reader uses status, compared with only 0.2 pp for the similarity-based reader. This 5.5 pp interaction suggests that the record is useful mainly when the reader actually uses the stored status information.

Critical weaknesses
  1. I think the deterministic guarantees are stated more strongly than the implementation supports. Appendix A keeps a proposition even when its supporting span fails the containment check, and substring matching by itself does not establish that the span entails the proposition. The model also decides vertex types and keys. As Proposition 4 notes, duplicate keys can leave two current values that refer to the same real-world attribute. In addition, the disclosure rule is ultimately followed by the reply model rather than enforced entirely by the store. For these reasons, I do not think the statement that “none of the three depends on the model following an instruction” is justified as currently written.

  2. The AI-LieDar result does not yet isolate the effect of the proposed structural mechanism, and I also have concerns about the standard deviations in Table 1. Verified facts are introduced through labelled initialization, while none of the 620 maintainer-written vertices is verified (§3.3). The provenance ablation also removes reply-side rules, whereas the transcript baseline does not receive the same labels. A baseline with matched labels and disclosure instructions seems necessary to separate the effect of the memory structure from the extra supervision.

  3. There is also a numerical issue in Table 1. Under the stated setup of 60 scenarios and three runs, 7 of the 13 reported True% standard deviations, including 53.3 ± 2.6, do not appear to be attainable from integer success counts under the usual sample or population SD definitions with the reported rounding. I may be missing an evaluation detail, but the paper should provide the underlying per-run counts and explain exactly how these statistics were computed. Until these two issues are clarified, I do not think the reported 95.0% result is sufficient evidence for the structural-enforcement claim.

Questions for authors
  1. Could you provide the truthful numerator and denominator for each run of every AI-LieDar configuration in Table 1, together with the exact standard-deviation and rounding procedure? If the effective denominator is not 60 in some cases, which scenarios were excluded and why? For the LoCoMo significance claim in the caption, please also report the paired conversation-level effects and the Holm-adjusted p-values.

  2. Could you add a transcript or existing-memory baseline that receives the same verified/hidden/unfavourable initialization labels and the same disclosure instruction as INCUMBENT? I would like to see how much of the AI-LieDar improvement remains once this supervision is matched.

  3. Table 11 shows lower AMemGym input-token usage for INCUMBENT, but the measured wall-clock time is 14.0 seconds per turn, compared with 8.9 seconds for AWE and 11.7 seconds for Pure LLM. Is the efficiency claim intended to refer only to input-token cost? It would also help to explain the source of the latency increase. Since Section 2.3 already notes that the practical write cost depends on store size, the efficiency discussion should be scoped accordingly.

2 comments

To Reviewer fqrM

Thank you for verifying our numbers directly and for suggesting concrete experiments to isolate our claims. We address each of your points in turn.

[W3 / Q1] Standard deviations and per-run counts in Table 1

As you noted, there was an error in the AI-LieDar standard deviations reported in Table 1. The reported subscripts were computed from an earlier development execution rather than from the final three runs. After the final runs were completed, we updated the means in Table 1 but inadvertently failed to update the subscripts. We verified against our revision history that those subscripts matched the means from that earlier draft. This is why, as you rightly observed, 7 of the 13 reported standard deviations could not be obtained from three integer counts out of 60 alongside the final means.

To clarify the relationship between the earlier execution and the final three runs: the earlier execution was a development run conducted to check the pipeline before the evaluation settings were finalised. Every main number reported in the paper comes from the three runs executed after fixing the units, tests, and margins in Appendix G and the hyperparameters in Appendix D. Those three runs were not selected post hoc; they are the entirety of the runs executed after freezing the configuration, and no post-freeze runs were excluded. Apart from the stale subscripts in Table 1, no numbers from the development run were used anywhere in the paper.

Each of the final three runs includes all 60 scenarios. Whenever an API call failed transiently, it was retried within the same run, so no scenarios were dropped and the effective denominator is 60 in every run.

Recomputing from the episode-level labels of the final three runs yields the values below. The recomputed means for True %, False %, and TUS match Table 1 in every cell; only the standard deviations change.

BackboneAgentRun 1Run 2Run 3Total / 180True %Reported SDCorrected SD
gpt-5.4-miniPure LLM2327257541.72.43.3
RAG2831288748.32.92.9
A-MEM2826308446.73.13.3
AWI3331329653.32.61.7
Mem0-G2933309251.13.33.5
AWE2727247843.32.82.9
INCUMBENT57565817195.01.71.7
gpt-5.4Pure LLM34373410558.32.62.9
AWE41383811765.02.92.9
INCUMBENT58595717496.71.41.7
Qwen3.5-9BPure LLM1511133921.73.43.3
AWE7862111.73.01.7
INCUMBENT49494614480.02.92.9

Each standard deviation is the sample standard deviation (n−1n - 1) across the three per-run truthful percentages (count/60×100\text{count}/60 \times 100), rounded to one decimal place. To check whether the same issue affected any other columns, we recomputed all standard deviations in Table 1 (TUS, AMemGym, and LoCoMo) under the exact same procedure and confirmed that no other columns were affected.

This correction does not affect any conclusion of the paper, because the descriptive Wilson intervals in the text and all significance tests are computed at the scenario level rather than from run-level standard deviations (Appendix G).

For the LoCoMo significance claim, the requested conversation-level paired effects (INCUMBENT−Pure LLM\text{INCUMBENT} - \text{Pure LLM}, n=10n = 10) and Holm-adjusted pp-values are as follows:

ColumnINCUMBENTPure LLMDifference [95% CI]Holm pp
Single72.467.7+4.7 [2.3, 7.1].012
Multi59.654.8+4.8 [1.1, 8.5].064
Temporal77.368.7+8.6 [5.1, 12.1].006
Open26.723.5+3.2 [−2.9, 9.3].33
Mean59.053.7+5.3 [3.2, 7.4].005

The per-conversation Mean differences are [8.6,2.5,6.7,1.4,9.7,4.4,6.1,0.3,10.4,2.9][8.6, 2.5, 6.7, 1.4, 9.7, 4.4, 6.1, 0.3, 10.4, 2.9], with INCUMBENT scoring higher in all 10 conversations. However, as you rightly questioned, the blanket statement in the Table 1 caption was imprecise: after Holm adjustment, p<.01p < .01 holds for the Mean and Temporal columns, whereas Single is significant at p<.05p < .05 and Multi and Open are not statistically significant due to higher conversation-level variance on categories with fewer items. We will correct the caption accordingly. Our main-text claim on LoCoMo concerns Mean F1 and remains unchanged.

[W2 / Q2] Baseline with matched initialization labels and disclosure instructions

We agree with your concern and ran the exact baseline you requested. We took the verified / unfavourable / hidden labels produced by INCUMBENT's initialization call, attached them directly to the corresponding sentences of the private record in the Pure LLM baseline, and provided the identical reply disclosure rules.

ConditionTrue %
Pure LLM41.7
+ reply rules56.7
+ reply rules + initialization labels70.0
INCUMBENT95.0

Matching both the labels and the instructions accounts for 28.3 pp of the 53.3 pp total difference, while a 25.0 pp improvement remains (95% CI [16.2,33.8][16.2, 33.8], p<.001p < .001). We also wish to clarify one point about the setup: at initialization, INCUMBENT receives the exact same unlabelled private record as the baselines (Table 4); the difference lies in the initialization prompt that structures it. The third row above gives the baseline the output of that initialization prompt directly on its private record.

Your observation that the existing provenance ablation in Table 7 (61.7%, which removes the source field alongside the rules that read it) confounds fields and rules is also well taken. To separate them cleanly over the same stored graphs, we crossed the presence of the reply rules with masking all store-assigned labels (source, valence, status):

Rules keptRules removed
Labels kept95.065.0
Labels removed68.360.0

The rules improve truthfulness by 30.0 pp when labels are present, compared with 8.3 pp when labels are masked (interaction +21.7+21.7 [9.8,33.6][9.8, 33.6], p=.001p = .001). This mirrors the structure of Table 2: stored metadata pays when the reader is bound to it.

[W1] Scope of the deterministic guarantees

We agree with your critique. The sentence in Section 1 overstated what the store guarantees on its own. We revise it as follows:

“Each guarantee is a property of how a write is applied or how a read is selected, and holds for every delta the model proposes. What the model proposes remains its own decision: the type and key of a vertex, whether a span supports its proposition, and whether the reply follows the disclosure rule.”

We will revise the corresponding sentences in the abstract and Section 2.1 in the same way. The invariants enforced by the store remain unchanged, as do all empirical results: every stored span exists verbatim in the transcript, no maintainer write can mint verified provenance, each key holds at most one current value, and the reader is bound to the value in force. We measured the empirical error rates of the decisions that remain with the model across a 900-vertex sample and report them in detail in our response to Reviewer a8dv ([W1 / Q1]): 3.3% of verified spans fail to support their proposition, 2.0% of facts are misclassified inferences, and 2.8% of hypotheses are misclassified user statements, bounding the impact on AMemGym Overall at a maximum of 1.5 pp. Adherence to the disclosure rule on AI-LieDar is shown in Table 1 and Figure 3a: out of 180 episodes, 0 are judged false and 9 are judged partially truthful.

[Q3] Scope of the efficiency claim and source of AMemGym latency

Yes, our efficiency claim refers specifically to input tokens, and we will state this explicitly in the text. The higher wall-clock latency on AMemGym (14.0s vs. 8.9s for AWE) comes from autoregressive output generation by the maintainer. Both agents update every two turns (κ=2\kappa = 2) and make a comparable number of LLM calls per turn (1.5 for INCUMBENT), but INCUMBENT generates structured graph deltas together with reasoning tokens, averaging 1,150 output tokens (including reasoning) per call compared with 760 for AWE. As you suggested, we also scope the efficiency discussion explicitly: the input-token advantage is measured at the store sizes of these benchmarks, and write cost grows with the number of attributes in the store (on AMemGym, attributes grow 3.0×3.0\times over a conversation while the transcript grows 11×11\times; Appendix L).

We hope these clarifications and additional experiments resolve your concerns, and we welcome any further questions.

Reviewer_8n74OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes INCUMBENT, a structured memory system that separates LLM-generated memory updates from deterministic store-level enforcement. It organizes facts, concerns, and hypotheses into attribute-based hyperedges, maintains provenance and historical states, and uses status-aware retrieval to avoid outdated information. Experiments on AI-LieDar, AMemGym, and LoCoMo demonstrate improvements in truthfulness, memory accuracy, and retrieval efficiency.

Critical strengths
  1. Strong empirical results. The method achieves substantial improvements across three benchmarks with reduced input token consumption. The evaluation includes multiple backbones, repeated runs, and statistical significance tests.

  2. Clear motivation and informative ablations. The separation between LLM proposals and deterministic memory maintenance is well motivated. In particular, the ablation of supersession records and status-aware retrieval effectively demonstrates the importance of enforcing state changes during retrieval.

Critical weaknesses
  1. Limited novelty and insufficient comparisons. The core pipeline of extracting, updating, and retrieving entity attributes overlaps with existing structured and temporal memory systems, such as Zep/Graphiti and APEX-MEM. Without comparisons to these methods or simpler versioned attribute stores, the additional value of the proposed framework remains unclear.

  2. Dependence on LLM-based attribute identification. The currency guarantee holds within each canonical attribute, but identifying and merging semantically equivalent attributes still relies on LLM decisions. The proposed mutual-nearest-neighbor alias repair is insufficiently motivated, leaving its practical reliability unclear.

Questions for authors

Q1. What are the key differences between INCUMBENT and existing temporal memory systems such as Zep/Graphiti and APEX-MEM? Could simpler versioned attribute storage achieve similar benefits?

Q2. Since memory is updated incrementally, why use mutual-nearest-neighbor alias repair instead of matching newly extracted attributes against existing ones during updates?

Q3. Why are the reported AMemGym results for AWE and Mem0-G substantially lower than those in the original benchmark paper? Are there stronger existing baselines on AMemGym or LoCoMo that could be compared under the same evaluation setting?

Q4. What roles do open concerns and inferred hypotheses play in the system? Why are they stored as persistent memory, and how do they contribute to the evaluated tasks?

3 comments

Thank you for highlighting the strengths of our empirical results and ablations, and for asking how INCUMBENT relates to recent temporal memory systems. We address each of your questions below.

[W1 / Q1] Comparison with existing temporal memory systems (Zep/Graphiti, APEX-MEM) and simpler versioned stores

The core difference between INCUMBENT and existing temporal memory systems lies in who makes the two decisions when a value changes: (i) whether a prior value has been superseded, and (ii) what prevents a superseded value from being retrieved as current. We verified Graphiti from its source code (v0.31.2) and APEX-MEM from its paper:

SystemWho decides whether a value is superseded?What prevents a superseded value from being read?
Mem0(-G), AWEModel (UPDATE / DELETE)Nothing
GraphitiModel identifies candidates; date comparison finalisesNothing; default search returns edges alongside invalid_at
APEX-MEMNot decided at write time (append-only)Retrieval agent at query time
INCUMBENTStore (Proposition 2)Store (Proposition 3)

Regarding whether a simpler versioned attribute store would suffice, the question can be read in two ways, and our ablations measure both:

  • A store that keeps version history on attributes but ranks by similarity at read time corresponds directly to the “record kept, similarity reader” cell of Table 2: it reaches 0.429 Overall on AMemGym, where keeping the version record adds only 0.2 pp (statistically equivalent to zero).
  • A store that always reads the latest version per key implements INCUMBENT's currency rule. However, as long as the model chooses the keys, per-key currency alone is not sufficient; each of the remaining components makes a separate, statistically significant contribution (full current-value visibility for the maintainer: +3.1 pp on AMemGym; source-span grounding: +8.4 F1 on LoCoMo; store-enforced provenance: +33.3 pp on AI-LieDar; Table 7).

Following your suggestion, we are running Graphiti on AMemGym under the exact same configuration as the other memory agents (top-30 retrieval, 4 recent messages in context). Because Graphiti incurs a very high write cost (shown below), the full 20-user evaluation could not finish within the rebuttal window. We therefore report the complete results on the first user (80 user turns, 110 questions) as they stand. Because a single user cannot support a significance test, we do not claim overall performance superiority from this table, and we will post the full 20-user results for the best-performing Graphiti configuration (user turns only) to this thread as soon as the run finishes.

Same AMemGym userCorrect / 110Write LLM callsWrite input tokens
Graphiti, library default (both speakers)341,0745.18M
Graphiti, default + entity-summary reading37(same stored state)
Graphiti, user turns only444431.07M
AWE41770.39M
Pure LLM45——
INCUMBENT54400.17M

This execution makes two structural differences concrete in the numbers:

  1. Who decides state changes. Out of Graphiti's 1,074 write LLM calls, 821 (76%) are edge-invalidation/deduplication calls asking the model whether a new edge duplicates or contradicts existing edges. Even when restricted to user turns only, 215 of 443 calls (49%) are spent on this check. INCUMBENT makes zero such calls because supersession is an unconditional effect of writing a state fact to an occupied key (Proposition 2).
  2. The unit of storage. A fact in Graphiti must be a relational edge between two distinct entities, and its extraction prompt instructs the model to skip details that lack a second entity. Moreover, because it always extracts the speaker as an entity, feeding both speakers causes the assistant's long replies to dominate the graph (461 nodes, 1,429 edges). Feeding user turns only shrinks the graph to 208 nodes and 446 edges and raises accuracy from 34/110 to 44/110. By contrast, a hyperedge in INCUMBENT represents a single attribute of a person and does not require a second entity.

Under its most favourable setting, Graphiti performs on par with Pure LLM on this user, consistent with our observation in Section 3.2 that existing memory agents do not surpass an agent that holds the raw transcript. For APEX-MEM, no public implementation is available, so we compare at the design level; the 88.88 LoCoMo score reported in that paper is an LLM-as-a-Judge accuracy on a different backbone and is not directly comparable to our token F1.

[Q3] Why AWE and Mem0-G scores on AMemGym differ from the original benchmark paper

The difference comes from two factors:

  1. Evaluation protocol (off-policy vs. on-policy). To ensure every agent ingests the exact same turns and answers from the exact same conversation history, we evaluate all agents by replaying the fixed benchmark transcripts (Appendix E). In the terminology of the AMemGym paper, this is the off-policy setting. Table 2 of the AMemGym paper reports both: for AWE, the on-policy memory score is .291 while the off-policy score is .253 (the same protocol distinction applies to Mem0-G). The higher numbers you noted are the on-policy figures.
  2. Backbone model. The original benchmark paper used gpt-4.1-mini, whereas our main table uses gpt-5.4-mini. Notably, Pure LLM (which reads the raw transcript) scores .198 off-policy in the original paper and .206 in our Table 1. We will make this off-policy distinction explicit in Appendix E.

To verify that the remaining gap is not an implementation artefact, we ran our AWE and INCUMBENT implementations on the original paper's backbone (gpt-4.1-mini):

gpt-4.1-mini, AMemGym Memory ScoreOriginal paper (off-policy)Our implementation
AWE.253.247
INCUMBENT—.340

Regarding stronger existing baselines: AWE is the strongest memory agent reported in the AMemGym paper and is included in our comparison. The only system evaluated in the AMemGym paper not in our Table 1 is Nemori (.231 on-policy, below AWE), and we have now added Graphiti above (for recent LoCoMo baselines such as APEX-MEM, please see [W1 / Q1] above).

[W2 / Q2] LLM-based attribute identification and mutual-nearest-neighbour alias repair

As you noted, key identity is decided by the model and our currency guarantee holds per canonical key (Proposition 4). Matching newly extracted attributes against existing ones during updates is in fact our primary mechanism: at every maintenance call, the maintainer is shown the current value of every attribute in the store (Equation 3) and reuses existing keys directly. ALIAS-HYPEREDGE is a secondary repair mechanism that catches only the residual duplicates that slip past write-time reuse. Across 2,230 attributes, alias repair reduces the duplicate rate from 9.5% to 3.8%, and manual inspection of all 140 applied aliases confirmed that 127 (91%) are valid merges of synonymous attributes. We use mutual nearest neighbours because it requires no corpus-specific similarity threshold and is self-bounding, offering at most ⌊n/2⌋\lfloor n/2 \rfloor candidate pairs over nn hyperedges (Lemma 6).

[Q4] Roles of open concerns and inferred hypotheses

A hypothesis gives the model a dedicated bucket to record its own inferences so that they are never stored as user-stated facts. Hypotheses are not kept permanently: any hypothesis not touched again within W=8W = 8 utterances is automatically retired by the store, which clears more than half of all created hypotheses (Appendix A, Appendix J). A concern represents an unfulfilled user request and remains visible in the maintainer's view until it is addressed or superseded by a newer concern on that attribute.

Their empirical contributions to the evaluated tasks are as follows:

  • On AI-LieDar, the 620 maintainer-written vertices consist of 318 facts, 97 concerns, and 205 hypotheses. Ablating the hypothesis bucket so that inferences are stored as facts reduces truthfulness from 95.0% to 83.3% (p=.001p = .001).
  • On LoCoMo, 11% of the vertices selected by the reader to answer questions are concerns. However, none of the three benchmarks directly evaluates the fulfilment of open user requests over time, and we will note this explicitly as a limitation of the benchmark suite.

We hope these responses address your concerns, and we would be glad to answer any further questions.

Thank you for the detailed rebuttal. I appreciate the clarifications on the differences and experiment results from prior work, particularly regarding versioning and store-enforced updates, and the additional experimental evidence.

My main concern remains the methodological novelty. The components, including versioning, LLM-based update decisions, and temporal memory management, have been explored in prior work. While INCUMBENT introduces deterministic store-level enforcement, it still partially relies on LLM decisions for attribute identification and update semantics. I am not fully convinced that its contribution is sufficiently distinct from existing approaches.

Overall, I acknowledge the empirical results, but my concern remains. I am inclined to maintain my current rating, while remaining open to other reviewers' new perspectives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To All Reviewers

To All Reviewers

We sincerely thank the three reviewers for their careful reading and constructive feedback. All three highlighted the separation between model-proposed updates and store-level enforcement, together with the crossed ablation in Table 2, as key strengths of the paper. The concerns raised fall into three main areas; for each, we have carried out additional experiments and analyses reported below, and we clarify and tighten the corresponding statements in the paper:

  1. Scope of the guarantees. We agree that the original phrasing stated the deterministic guarantees more broadly than the implementation supports, and we have revised it accordingly. We also measured how often the decisions that remain with the model fail in practice across a stratified sample of 900 vertices (verified spans failing to support their proposition: 3.3%; inferences misclassified as facts: 2.0%).
  2. Interpretation of the AI-LieDar results. As requested, we added a baseline that receives the exact same initialization labels and disclosure instructions as INCUMBENT. Controlling for this supervision accounts for 28.3 pp of the 53.3 pp total gain, while a 25.0 pp improvement remains (p<.001p < .001). We also added a crossed 2×2 ablation separating store-assigned labels from reply rules.
  3. Comparisons with existing temporal systems and construction cost. We compare against Graphiti at the source-code level and report empirical results on AMemGym, explain the score difference from the original AMemGym benchmark paper, and report full LoCoMo memory-construction costs across all agents. In doing so, we explicitly scope our efficiency claims.

We also make three corrections: the run-level standard deviations for AI-LieDar in Table 1, the LoCoMo significance statement in the Table 1 caption, and the asymptotic write-cost bound in Proposition 5. All additional statistical tests reported in this response are Holm-adjusted separately from the pre-registered test family in the paper. All clarifications, corrections, and additional analyses will be incorporated into the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.