- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper studies whether an agent can tell, before it retrieves anything, that its memory store holds a record it should no longer use (for example, a cancelled plan). It calls this "negative obligation knowledge" and treats it as metamemory monitoring only, not control.
The reader sees a summary with no stored values: record counts by lifecycle state and lists of attribute names. Each of 148 questions is asked against three versions of one store that differ in a single attribute, giving 444 items, so the question wording can't give away the label.
Five LLMs are compared with a simple cosine lookup over the attribute names that makes no model call. On the short ground-truth lists (2.6 names on average), no model reliably beats the lookup. On lists padded to 60 names, only GPT-5.6 Luna and Sol stay ahead, and only Sol keeps a reliable lead once list length and counts are fixed.
The stronger models' extra edge comes from reading counts for retracted and expired records. A lookup given one count feature closes that gap within a question.
A census of four backends (FR-Bank, Mem0, A-MEM, Graphiti) finds that each one omits or misstates lifecycle information, and readers lose most of their gain under the store's own metadata. A retrieval-time fetch keyed on names reaches as many forbidding records without a model gate as with one.
Critical strengths
Clean design - The same question is asked of three store versions, so question-only leakage drops to chance by construction. This matters because the authors' own audit shows question text alone predicts the labels on the original question sets, so most earlier absolute scores in this area are suspect.
Strong baselines - The name lookup, count readers, random-token lookup, sibling-name control, padding, paraphrase check, moved-count check, and equivalence tests make the negative result believable rather than a weak-baseline story. It is also an unusually transparent record: predictions registered before runs, failed predictions reported (34 of 106), and run deviations disclosed.
Critical weaknesses
The main result is close to built in, because the authors' summary format lists the target attribute under the superseded heading, so a similarity match is nearly the answer. "Models don't beat the lookup" mostly restates how the input was built.
The more interesting findings rest on small or thin evidence. The 60-name result holds for one model family only, and Luna's lead doesn't survive the equal-length sibling control. The counted-state advantage comes from 30 retraction and 30 expiry pairs and disappears against a lookup with one count feature.
External validity is narrow, since everything uses one synthetic bank of 40 personas tied to FR-Bank, with no second benchmark and no real user data.
There is no downstream evidence that better monitoring reduces stale answers, which is the motivating problem. Control is explicitly out of scope, and the answer-stage analysis is exploratory: a token detector whose human labels weren't collected at submission.
The statistics and reporting are hard to trust and follow. There are 71 main-text comparisons with no multiplicity correction, the post hoc matched metric decides several claims, and run settings vary across rows (draw counts, temperatures, agent harness vs raw API). The main rubric scores stale presence rather than the intent-conditional obligation in Definition 1, and 94 pages with more than 100 tables make the core story very hard to extract.
Questions for authors
Given that the summary lists the target name directly, what result would have counted as a model "adding something"? Is there a summary design where the lookup is not near-ceiling by construction?
How do the main claims hold up after a multiplicity correction across the 71 comparisons, with the matched metric treated as secondary?
Under the intent-explicit rubric, which matches Definition 1, do the model-vs-lookup conclusions on the full counterfactual set change for all five models, not only mini and Luna?
Can you report the human-validated answer-stage labels, and without them, is there any evidence that a better obligation judgment changes final answers?
For future work, consider transfer to a second benchmark or real logs, intent questions without explicit cues, and a cost-aware comparison of model gating vs an always-on name fetch.
Update (17 Nov): new results and one correction
Dear Reviewer 8SKY,
We have three new results on your two questions: what is new, and whether the findings generalize.
1. The main idea now shows up in final answers.
Our main idea: whether an agent can use lifecycle information depends on how the store writes it. To test this, we kept everything fixed (stores, fetch, answer model) and changed only the lifecycle tags.
Why? FR-Bank marks many current records as outdated. So the warning fires on 119 of those 174 questions, against 6 with reference tags. The same fetch helps or hurts depending only on what the store wrote. Acting on tags helps only when the store writes them accurately.
2. Human labels confirm the answer results.
Two annotators who are not authors labelled 297 answers without knowing which method produced them.
3. A correction on Wikidata.
On 11 Nov we said that on 221 Wikidata pairs, the models did no better than the name lookup. Those pairs were mislabeled: even the "clean" version still held an outdated fact of the same kind, such as a past employer. So a tie was expected, and we withdraw that claim.
Also: a small fix to how ties are counted changes printed numbers by at most 0.033. No conclusion changes.
We are glad to clarify anything before the discussion closes.
Best regards, The authors