acceptodds
Under review as a conference paper at ICLR 2027

Knowing the Store: What a Memory Backend Must Write Down Before an Agent Can Read It

Abstract

An agent with long-term memory can answer from a record it should no longer use, such as a plan the user later cancelled. We ask what a memory store must expose for an agent to know this before retrieving anything, and we score that judgment on its own, as metamemory monitoring. Readers see only a value-free summary: record counts by lifecycle state and a list of attribute names. Each of 148 questions is asked against three versions of one store that differ in one attribute, so wording cannot give the answer away. What a model adds depends on the length of that list. On the short lists the benchmark's ground truth produces (2.6 names on average), none of five language models reliably beats a cosine-similarity lookup over the names, and the three stronger ones are equivalent to it within 0.05. Under a control that fixes counts and list length, none is reliably above it and none is shown equivalent. On lists longer than the stores we tested write, padded to 60 names, the lookup loses 0.18; GPT-5.6 Luna and Sol lose 0.06 to 0.07 and lead it by 0.13 to 0.14, while the other three fall with it. The lead holds for Sol against a sibling padded to the same length and counts. Reworded to share no word with the names, the lookup loses 0.02 and one stronger reader edges 0.04 ahead. On the short lists the three stronger readers lead only where the summary counts a state without naming it, and one count feature closes that lead within a question, though not when items are pooled. The backends we tested omit or misstate this information: under FR-Bank's own metadata, GPT-4.1 mini, Haiku 4.5 and the lookup fall from 0.73, 0.82 and 0.75 to 0.60, 0.71 and 0.59. In these tests, what the store wrote down limited every reader we gave it to. All stores are synthetic; control, whether a better judgment yields a better answer, is left to a later study.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent memory, long-term memory, metamemory, memory lifecycle, knowledge updates, retrieval-augmented generation, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v8xLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies whether an agent can tell, before it retrieves anything, that its memory store holds a record it should no longer use (for example, a cancelled plan). It calls this "negative obligation knowledge" and treats it as metamemory monitoring only, not control.

The reader sees a summary with no stored values: record counts by lifecycle state and lists of attribute names. Each of 148 questions is asked against three versions of one store that differ in a single attribute, giving 444 items, so the question wording can't give away the label.

Five LLMs are compared with a simple cosine lookup over the attribute names that makes no model call. On the short ground-truth lists (2.6 names on average), no model reliably beats the lookup. On lists padded to 60 names, only GPT-5.6 Luna and Sol stay ahead, and only Sol keeps a reliable lead once list length and counts are fixed.

The stronger models' extra edge comes from reading counts for retracted and expired records. A lookup given one count feature closes that gap within a question.

A census of four backends (FR-Bank, Mem0, A-MEM, Graphiti) finds that each one omits or misstates lifecycle information, and readers lose most of their gain under the store's own metadata. A retrieval-time fetch keyed on names reaches as many forbidding records without a model gate as with one.

Critical strengths

Clean design - The same question is asked of three store versions, so question-only leakage drops to chance by construction. This matters because the authors' own audit shows question text alone predicts the labels on the original question sets, so most earlier absolute scores in this area are suspect.

Strong baselines - The name lookup, count readers, random-token lookup, sibling-name control, padding, paraphrase check, moved-count check, and equivalence tests make the negative result believable rather than a weak-baseline story. It is also an unusually transparent record: predictions registered before runs, failed predictions reported (34 of 106), and run deviations disclosed.

Critical weaknesses

The main result is close to built in, because the authors' summary format lists the target attribute under the superseded heading, so a similarity match is nearly the answer. "Models don't beat the lookup" mostly restates how the input was built.

The more interesting findings rest on small or thin evidence. The 60-name result holds for one model family only, and Luna's lead doesn't survive the equal-length sibling control. The counted-state advantage comes from 30 retraction and 30 expiry pairs and disappears against a lookup with one count feature.

External validity is narrow, since everything uses one synthetic bank of 40 personas tied to FR-Bank, with no second benchmark and no real user data.

There is no downstream evidence that better monitoring reduces stale answers, which is the motivating problem. Control is explicitly out of scope, and the answer-stage analysis is exploratory: a token detector whose human labels weren't collected at submission.

The statistics and reporting are hard to trust and follow. There are 71 main-text comparisons with no multiplicity correction, the post hoc matched metric decides several claims, and run settings vary across rows (draw counts, temperatures, agent harness vs raw API). The main rubric scores stale presence rather than the intent-conditional obligation in Definition 1, and 94 pages with more than 100 tables make the core story very hard to extract.

Questions for authors

Given that the summary lists the target name directly, what result would have counted as a model "adding something"? Is there a summary design where the lookup is not near-ceiling by construction?

How do the main claims hold up after a multiplicity correction across the 71 comparisons, with the matched metric treated as secondary?

Under the intent-explicit rubric, which matches Definition 1, do the model-vs-lookup conclusions on the full counterfactual set change for all five models, not only mini and Luna?

Can you report the human-validated answer-stage labels, and without them, is there any evidence that a better obligation judgment changes final answers?

For future work, consider transfer to a second benchmark or real logs, intent questions without explicit cues, and a cost-aware comparison of model gating vs an always-on name fetch.

Reviewer_8SKYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether LLM agents can identify outdated information in their memory before retrieval. The authors evaluate several LLMs and memory systems, finding that simple baselines can perform comparably to LLM-based approaches, while the quality and completeness of stored memory play an important role in the results.

Critical strengths

The paper addresses a timely and relevant problem in agentic memory systems, particularly how agents can identify outdated information. The authors examine some useful details, such as how labeling choices affect performance, and provide concrete observations from the experiments.

Critical weaknesses

Limited novelty of the main findings. One of the main conclusions is that what a memory system stores fundamentally limits what downstream models can infer or identify. However, this observation seems rather intuitive and does not provide much new insight. I would have expected a deeper analysis of when or why different approaches fail, or findings that go beyond what we already know about information availability in memory systems. Unclear writing and experimental descriptions. Several parts of the paper are difficult to follow due to vague references and insufficiently defined terms. For example, the conclusion states, "On a named state no model reliably exceeds the name lookup on the short list; the three stronger readers are equivalent to it within 0.05." It is unclear what metric the 0.05 refers to, or which models are considered the "stronger readers." There are several similar instances throughout the paper. Model names such as Luna, Sol, and Opus should also be accompanied by their specific versions. These issues make the experimental findings unnecessarily difficult to understand. Limited generalizability. The experiments rely on a single synthetic dataset with only 40 personas, which may not capture the complexity of real-world agent memory systems. It remains unclear whether the findings would hold across different datasets, memory architectures, or more realistic scenarios. Concerns about experimental consistency. Some experimental settings appear to differ across models, including the number of runs, sampling configurations, and API setups. Without a clearer explanation or controlled comparisons, it is difficult to determine how much these differences may have influenced the reported results.

Questions for authors

The experiments mainly rely on a single synthetic dataset. Could the authors consider evaluating on other memory benchmarks, such as MemBench (ACL 2025), LoCoMo-Plus (ACL 2026), or Memora (ACL 2026), and comparing with more recent memory systems like APEX-MEM (ACL 2026) and MemGAS (ICLR 2026)? This would help better understand how well the findings generalize. Some experimental settings seem to differ across models, including the number of runs, sampling configurations, and API setups. Could the authors clarify these differences and explain whether they might affect the comparisons? A more consistent evaluation setup would make the results easier to interpret. Beyond showing that the stored memory limits downstream performance, what do the authors see as the main new insight from these experiments? It would be helpful to understand what these findings tell us about designing better memory systems, beyond the relatively expected effect of memory completeness and quality.

2 comments
AuthorsOpenReview ↗

Update (17 Nov): new results and one correction

Dear Reviewer 8SKY,

We have three new results on your two questions: what is new, and whether the findings generalize.

1. The main idea now shows up in final answers.

Our main idea: whether an agent can use lifecycle information depends on how the store writes it. To test this, we kept everything fixed (stores, fetch, answer model) and changed only the lifecycle tags.

  • Reference tags (clean labels): the revised fetch cuts stale answers from 38 to 23.
  • FR-Bank's own tags: the fetch does no better than plain retrieval (24 stale answers each). It also costs correct answers: on the 174 questions with nothing stale to avoid, correct answers fall from 53 to 43.

Why? FR-Bank marks many current records as outdated. So the warning fires on 119 of those 174 questions, against 6 with reference tags. The same fetch helps or hurts depending only on what the store wrote. Acting on tags helps only when the store writes them accurately.

2. Human labels confirm the answer results.

Two annotators who are not authors labelled 297 answers without knowing which method produced them.

  • They agreed with each other on 94% of answers, and with our automatic judge on 92%.
  • On their labels, the revised fetch cuts stale answers from 34 to 19 at three records, and from 30 to 19 at five. Both survive correction for multiple comparisons.

3. A correction on Wikidata.

On 11 Nov we said that on 221 Wikidata pairs, the models did no better than the name lookup. Those pairs were mislabeled: even the "clean" version still held an outdated fact of the same kind, such as a past employer. So a tie was expected, and we withdraw that claim.

  • The registered result on the other 79 pairs stands: the lookup's win rate is 0.92 (chance 0.50).
  • After relabeling (post hoc), the lookup still scores highest on pooled AUROC: 0.92, against 0.88 for mini and 0.85 for Luna.

Also: a small fix to how ties are counted changes printed numbers by at most 0.033. No conclusion changes.

We are glad to clarify anything before the discussion closes.

Best regards, The authors

Dear Reviewer 8SKY,

The discussion closes in about three hours, so this is a brief final note. Our replies above answer your questions on what is new, other benchmarks, settings and writing, and the revised PDF includes them.

We'd be super grateful if you let us know if we answered your concerns! Thank you again for your review.

The authors

Reviewer_cYPjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates whether agents can judge memory coverage and identify content requiring suppression before retrieval, using lifecycle counts and attribute names. Counterfactual store variants, reader comparisons, and backend audits show that simple lookup is competitive on short summaries, while some stronger models benefit from counts and longer lists. The study provides diagnostic evidence about metadata quality and reader behavior; downstream answer improvements remain unestablished.

Critical strengths
  1. The counterfactual design holds questions fixed while changing memory states, isolating summary information from question-wording cues. Additional name, count, and paraphrase controls strengthen the analysis.

  2. Inexpensive baselines and transparent reporting make the findings informative. The paper carefully distinguishes significance from equivalence and reports failed predictions, incomplete experiments, and inconclusive downstream results.

Critical weaknesses
  1. Measurement does not consistently match the NOK definition. The main rubric scores stale-content presence, whereas Definition 1 concerns suppression under the query’s intent. Appendix E.3 identifies 44 PRESENT questions with non-active records under their target attribute. The explicit-intent rerun in Table 28 covers only two models on the counterfactual set, leaving the full obligation-set and backend conclusions insufficiently validated under the intended target.

  2. Backend attribution remains uncertain. A-MEM’s unresolved links were not checked against the reference implementation, and FR-Bank’s lifecycle disagreements were measured against a manifest-to-record alignment without human adjudication (Table 3). Although disclosed, these limitations leave integration failures and reference errors insufficiently separated from backend-writing deficiencies.

  3. Independent generalization is untested. All experiments reuse the same 40 synthetic personas, including the Graphiti evaluation (§9; Appendix G.4). Paraphrase and padding controls perturb this same population. Because reader-versus-lookup performance changes substantially with summary composition (§5.2), replication on independently constructed memory histories is needed to establish the broader scope of the findings.

Questions for authors
  1. Do the full obligation-set and backend conclusions hold under an intent-consistent rubric, particularly for the 44 PRESENT questions identified in Appendix E.3?

  2. What implementation checks or adjudicated examples establish that the A-MEM and FR-Bank findings reflect backend deficiencies rather than integration or reference-alignment errors?

  3. Does lookup competitiveness replicate on independently constructed memory histories with different lifecycle-event and summary-length distributions? Such validation would materially strengthen the generalization claim.

4 comments
AuthorsOpenReview ↗

Update (17 Nov): new results on your questions, and one correction

Dear Reviewer cYPj,

Since our last reply we ran new tests on your questions about backend attribution and independent histories. The blind adjudication you asked for is being labelled now, and we will post its result in this thread within a few hours. Here is everything else.

In short: FR-Bank's own tags mark current records as outdated, which points to the backend; the lookup's drop on independent personas is explained by their mix of lifecycle states and recovers when every state is named; human labels confirm our answer judge and the stale-answer cut with reference tags; and we correct one Wikidata claim.

Q2. Is the FR-Bank loss a backend problem or a reference problem?

We kept the stores, the questions and the answer model fixed and changed only the lifecycle tags: reference tags against the tags FR-Bank writes itself.

  • With identical retrieved records, FR-Bank's tags alone make plain retrieval lose 24 of 75 correct answers on the 174 questions with nothing stale to avoid.
  • Where the record holding the answer can be identified, FR-Bank had tagged it as outdated in 10 of 11 lost cases.
  • In one case FR-Bank marks one of the user's two current-gym records as superseded and leaves the old gym active, so the answer names the old gym.

In these cases the store marks a current record as outdated, which points to the backend rather than the reference.

The adjudication. It is built and registered before any label: 75 entries, including the 30 that FR-Bank marks non-active and the reference keeps active, plus control entries, with the source of every call hidden. The adjudicator is not an author and has no knowledge of FR-Bank. We registered in advance both the prediction and a check that the evidence shown is sufficient to recognise a real change.

Q3. Does lookup competitiveness replicate on independent histories?

  • Additional personas. On 20 additional personas written by a different model, the lookup scores 0.554 pooled AUROC, against 0.739 on the original questions. The drop comes from the mix of lifecycle states. Within each state the lookup scores about the same in both sets: 0.911 against 0.954 on superseded records, which the summary names, and low on retracted and expired ones, which it only counts. The additional personas have far fewer supersessions (38% against 61%) and far more expiries (41% against 14%). When the summary names every non-active state, the lookup reaches 0.712 on them, against 0.730 on the original questions. This analysis is post hoc and descriptive, and the labels for these personas are still being checked by hand.
  • A correction. On 11 Nov we said that on 221 Wikidata pairs the models did no better than the lookup. Those pairs were mislabeled: even the "clean" version still held an outdated fact of the same relation, so a tie was the correct result. We withdraw that claim. After relabeling, which we did post hoc, the lookup's pooled AUROC is 0.92, against 0.88 for mini and 0.85 for Luna. The registered result on the other 79 pairs stands (win rate 0.92).

Downstream answers

Your summary noted that downstream gains were unestablished. Two new results:

  • Human labels. Two annotators who are not authors labelled 297 answers without seeing which method produced them. They agreed with each other on 94% of answers (kappa 0.91) and with our automatic judge on 92% of judgments. On their labels, the revised fetch cuts stale answers from 34 to 19 at three records and from 30 to 19 at five; both survive correction for multiple comparisons.
  • The store's own tags. With FR-Bank's tags the same fetch gives no reduction (24 stale answers, the same as plain retrieval) and loses correct answers. The fetch helps only with accurate tags.

A small fix. Tied reader scores were not always counted as ties. Recomputing with exact ties changes printed numbers by at most 0.033; no conclusion changes.

Thank you again. Your questions about reference alignment and independent histories led directly to the tag test and the persona analysis above.

The authors

AuthorsOpenReview ↗

The adjudication result

As promised above, here is the blind adjudication.

An adjudicator who is not an author and has no knowledge of FR-Bank labelled 75 entries as still current or no longer current, blind to which source made each call. We registered the prediction and an evidence check before any label.

  • Our prediction holds. Of the 30 entries that FR-Bank marks non-active and the reference keeps active, 22 were labelled still current (the registered bar was 20), 7 non-active and 1 ambiguous.
  • The evidence check passes. Of the 30 entries the reference marks non-active, 25 were labelled non-active, including all 12 entries tied to a known change.
  • FR-Bank also misses changes. Of the 15 entries the reference marks non-active and FR-Bank keeps active, 13 were labelled non-active.
  • Controls. Of 15 entries both sources mark current, 12 were labelled current; of 15 both mark non-active, 12 were labelled non-active.

So most of FR-Bank's extra non-active marks are over-marks, and FR-Bank also misses real changes, which supports reading the loss under its own fields as a backend problem. Some of its extra marks are real: 7 of 30 here, though the adjudicator also labelled 2 of 15 agreed-current controls as non-active, so part of that is labelling noise.

Thank you for asking for this check.

AuthorsOpenReview ↗

A gentle followup :)

Dear Reviewer cYPj,

The discussion closes today, so this is a gentle reminder. Our replies above answer each of your three questions, including the blind adjudication you asked for, and today's revised PDF includes them. If anything is still unclear, we would be glad to answer before the deadline.

Thank you again for your review.

The authors

Dear Reviewer cYPj,

The discussion closes in about two hours, so this is a brief final note. We would be grateful to know whether our replies, including the blind adjudication you asked for, address your concerns.

Thank you again for your review.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.