- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper examines how agent memory benchmarks, mainly LongMemEval-V2 and LoCoMo, score questions about things that never happened. In these benchmarks, a memory system retrieves information from a long interaction history, a fixed language model answers questions using that information, and a scorer grades only the final answer. About a quarter of the questions contain a false premise (e.g., asking which button appears between two buttons that are actually adjacent), and the correct answer is to say that no such thing exists.
The authors argue that this scoring setup rewards systems for confidently saying “that doesn’t exist,” whether or not they actually checked. Two features of the scorer combine to cause this:
The scorer only checks whether the answer matches the correct one. It never looks at what information the system retrieved. So an answer of “that doesn’t exist” gets full credit even if the system never saw the relevant part of the history and was simply guessing. Guessing is never penalized. Answering “I don’t know” scores 0, and so does a wrong answer. A system that is unsure therefore loses nothing by guessing “that doesn’t exist”: on a false-premise question the guess is right, and on a normal question it scores the same 0 that “I don’t know” would have.
Together, these mean the best strategy is to answer “that doesn’t exist” whenever the system is unsure, regardless of what it actually remembers.
To show this, they add a single paragraph to the retrieved context, stating that the history is complete and anything not listed doesn't exist. This raises accuracy on these questions from 16% to 72%, and even a system with no memory at all reaches 57%. This introduces a new failure which the score doesn't directly show: the system now wrongly denies the existence of things that do exist far more often (55 times instead of once).
The authors propose three fixes. They are:
- pairing each false-premise question with a matched normal question;
- standardizing the instructions given to the answering model;
- a scorer that checks each denial against the actual interaction history.
Critical strengths
S1. Thorough and well-controlled experiments. The central claim, that the benchmark's false-premise questions can be scored highly without using memory, is supported by several independent lines of evidence:
- the same fixed answer given to every question, with no memory and no model reading anything, scored under four judge models from four vendors (Table 1);
- a comparison of no memory vs. real retrieval, each with and without the added paragraph (§3.2);
- a breakdown of which part of the added paragraph earns the credit (Table 3).
The appendix anticipates the usual objections to a critique paper and performs several ablations which strengthen the overall claims.
Critical weaknesses
W1. Limited novelty and undemonstrated practical impact. The core mechanism is a known one, and the authors identify it themselves: Kalai et al. (2025) showed that when a wrong answer and an "I don't know" score the same, guessing is always the better strategy (L092–093). This paper applies that observation to one memory benchmark, where the guess is "this doesn't exist."
The paper's most novel component, a check of whether each "doesn't exist" answer was backed by evidence in the interaction history, is by the authors' own account a prototype. It covers only 92 of the 128 relevant questions and cannot be applied to the second benchmark (L483–484).
The paper's second benchmark (LoCoMo) adds little. Its false-premise questions were already known to be unusable: the benchmark's own evaluation code crashes on 444 of these 446 questions (L2300–2301), and prior audits note that these questions are usually left out of reported scores (L2286–2288). Showing that a part of a benchmark that is already broken and already ignored can also be gamed tells us little that is new.
Most importantly, the paper does not show that the problem affects any published result. The authors state that they "do not claim that published rankings are inflated" (L471). The one reference system they reran gained nothing detectable from its built-in hint, and the top-ranked system was not tested. The paper therefore shows that the score can be inflated, but not that this changes any conclusion drawn from the benchmark.
W2. The writing makes a simple finding hard to understand and verify. The paper's core idea is simple: the benchmark gives credit for answering "this doesn't exist" without checking whether the system had any evidence for that answer. Yet the paper is very difficult to follow. It introduces many paper-specific terms and reports several numbers and confidence intervals in nearly every sentence, so the main argument is buried under detail. Even with a background in this area, I needed multiple careful passes to reconstruct how answers are scored and what each reported percentage measures.
The main text also does not stand on its own. It contains over 40 references to the appendix, and understanding several central claims, including how the paper's main proposed fix works, requires reading the appendix. This matters because a critique paper's value depends on readers being able to check its claims, and the current presentation makes that unusually hard.
I would encourage the authors to build the main text around one worked example, define each term before using it, and keep only the numbers essential to the argument in the main paper.
Questions for authors
None
4 comments
Response to RHJX (2/2): W1a–W1c, an error penalty cannot remove a correct guessed denial, only the recording check grounds it, and LoCoMo still counts it; W2, the rewrite
W1a. "The core mechanism is a known one." We agree that the incentive is the one Kalai et al. identify, and the paper cites them for it (lines 092–093). The core difference is what goes wrong: Kalai et al. concern wrong answers that binary grading fails to penalise; on a false-premise question the guessed denial is correct, so an error penalty cannot remove its credit. Rescoring all 451 questions under Kalai et al.'s penalty of t/(1−t) per wrong answer:
| Baseline, without and with the note | Correct | Wrong | Note's gain at a penalty of 1 / 3 / 9 |
|---|---|---|---|
| All 451 questions | 155 → 233 | 208 → 204 | +82 / +90 / +114 (net score) |
The penalty does charge the note on the answerable questions, where wrong answers rise from 139 to 170 and false denials from 1 to 55; on the false-premise questions it cannot, because there the note's denials are correct: correct answers rise from 20 to 92, while wrong answers fall from 69 to 34 and UNKNOWNs from 39 to 2. What is missing from those denials is evidence, which no error count sees; of the 92 credited denials, the recording check accepts 4 (71 of the 92 fall on questions it covers).
The setting also differs from the one Kalai et al. study. In Kalai et al., the model guesses on its own, so a confidence target stated in its instructions reaches the guesser. Here the system under test writes the reader's input (lines 042–046), and the note supplies exactly the confidence a target asks for: a reader told to distrust completeness claims in the memory context still credits the note with +48 points (Table 4).
W1b. "The paper's most novel component ... is a prototype."
- Novelty. Of the twelve published treatments of false-premise questions we audited, each either drops these questions or scores the answer alone; none reads what the reader was given (App. H). A positive answer can be traced to one passage; a denial needs evidence that the reader saw everything the recording holds on the point, which is what the check tests, and it passes the known-answer test of W1d.
- Coverage. LongMemEval-V2 does not publish each question's page, so we annotated pages and confirmed 92. 30 of the other 36 are settleable in principle: 5 lack a confirmed page in our annotation, and 25 compare two forms or span several pages, which needs the benchmark's page records and a two-page check we have not implemented. For an estimated 19 of 128 (95% interval 12–28; App. J), no list can settle the denial; we score those by pair-conditioning and denial precision.
- Second benchmark. On LoCoMo, requiring the annotated evidence session removes all credit of a no-memory system (100 to 0) but leaves BM25 at 86.8 of 87.9: necessary, not sufficient. The full check needs a per-question scope list only the benchmark can write; we have not run it.
W1c. "The paper's second benchmark (LoCoMo) adds little." We agree that the defect is known; what the paper adds is the mechanism, and published systems still count these questions. The crash at lines 2300–2301 concerns only the official forced-choice construction. Four of the ten published LoCoMo treatments we audited count the false-premise questions in their headline score or ranking (A-Mem, MAGMA, REMem, HiGMem; App. H), and in REMem's and MAGMA's tables they decide three orderings (W1d). On LoCoMo the answer instruction, not memory, sets the score: in our end-to-end rerun, Mem0 scores 42.6 with no instruction, 0.0 with the prompt of its own evaluation code, and 96.9 with one instruction shared by all systems; released scorers order the same pair of conditions oppositely in 46–55% of the combinations we replayed (App. M.2).
W2. "The writing makes a simple finding hard to understand and verify." We agree, and the revision follows your suggestion. §1 is built around one worked example, drawn in Figure 1a:
The question asks which action lies between "Reorder" and "Ship" on the toolbar of an order page; the gold is a denial, because the two are adjacent. The released baseline retrieves a different page, so the reader answers UNKNOWN and scores 0. With the note added to the same memory context, the reader denies and is credited, although the toolbar never reached it. The recording check finds no complete toolbar list in the memory context and does not accept the denial.
Terms are defined in a short list at the start of §2, and coined terms are replaced (false-premise question for denial question, recording check for licence, source question for twin). §4.3 explains the recording check in Algorithm 1, and the main text points to the appendix 15 times instead of 45. Changes are marked in blue.
Thank you again for your careful review. If these answers address your concerns, we would be grateful if you would reconsider your assessment.
I thank the authors for their rebuttal. Prior to the rebuttal, my soundness score of the paper was “3: I have no substantive concerns about the paper’s soundness,” with no questions to authors and it remains the same.
The paper is a critique paper of a popular benchmark. It’s rare to have such papers, and it’s difficult to evaluate their contributions and recommend a rating.
Critique papers are useful and should be encouraged in AI research; they show us common pitfalls to avoid. Making benchmarks is common, but making a good benchmark is hard, and sometimes entire research directions are built on the foundations of a benchmark (ARC-AGI being a prominent example).
This paper focuses on LongMemEval-V2, which is also quite a popular benchmark. Agent memory has become a big subfield, with many startups working to improve agents’ memory systems and millions of dollars being spent on it. LongMemEval-V2 is a central part of this subfield.
I don’t have any substantial concerns. The writing in the original submission was unclear, but the revised sections I checked have improved. In light of the rebuttal, I am raising my score from 2 to 3.
At the same time, it’s hard for me to get excited about the work, which is what keeps me from a higher score. The scope of the paper is very narrow: authors have shown 3 examples of wherein the memory system ranking has been changed because of the issue they find in the benchmark. But those are point estimates and the gap seems quite small too. Rerunning them with multiple model backends and providing confidence intervals might not show the ranking ordering to be swapped.
Since critique papers are difficult to assess with standard criteria, and the reviewer scores cluster around borderline without substantive technical objections, I recommend that the AC look at the submission holistically rather than focusing entirely on our scores, and make an appropriate decision based on their experience in research and reviewing.
I am happy to answer any questions that the authors or AC may have.
Thank you for your guidance
Dear Reviewer RHJX,
Thank you very much for reading our rebuttal and for your updated assessment. We are sincerely grateful for the time and care you have devoted to our paper. Your guidance on presentation, in particular your suggestion to build the main text around one worked example and to define each term before use, has made the paper much clearer. Your comment on whether the problem affects published results also guided us to connect our findings to published work, which has helped strengthen the practical relevance of the paper.
We are also grateful for your thoughtful remarks on critique papers and on the role of benchmarks in shaping a research direction. We sincerely hope our work can contribute to the agent memory community by helping build evaluations that reward what a memory system actually remembers.
Thank you once again for your generous and valuable guidance.
Best regards, The Authors
Response to RHJX (1/2): W1d, the false-premise questions already decide published comparisons
Thank you for the careful review, and for finding the experiments "thorough and well-controlled" and the central claim "supported by several independent lines of evidence". We label the four parts of Weakness 1 in the order of your review: W1a (Kalai et al.), W1b (the check as a prototype), W1c (LoCoMo) and W1d (published results). The note is our fixed text in the memory context stating that the recording is complete; the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:
W1d. "The paper does not show that the problem affects any published result." The weakness is visible in published results.
Removing LoCoMo's 446 false-premise questions reverses two published orderings in REMem's tables (ICLR 2026) and one in MAGMA's (ACL 2026), recomputed from the published per-category scores (which reproduce each overall score to within 0.1):
On the false-premise column, Graphiti ranks first of ten methods (83.0), above the oracle handed the gold evidence (70.6), while refusing 954 of 1,986 questions; by REMem's own refusal precision it is last of the three systems REMem measures (38.9%, against 73.3% for REMem and 40.0% for Mem0). We find no evidence that any system was built to game the score; the point is that the column deciding these comparisons cannot tell a grounded denial from an asserted one.
On LongMemEval-V2, the top reference system earns its premise-awareness gains through the channel we study. AgentRunbook-C is instructed to "hint at the downstream reader" in the memory context when the evidence contradicts the premise, and AgentRunbook-R's reranker to tell the downstream model to "call out the flawed premise"; the benchmark paper attributes AgentRunbook-C's abstention gains to this hint. The scorer cannot check "the evidence contradicts the premise", so it pays for the claim either way:
In our rerun, the corrections demote the credit that answer text buys. Under denial precision and pair-conditioning, a no-memory system with the note falls from second to last of five, and the two real memory systems keep their order. Only the recording check passes a known-answer test: on a complete right-page list, the same list with half its labels removed and a wrong-page list, which the denial score cannot separate (75.0, 70.3 and 71.1; not significant), it accepts 29, 17 and 0 of the credited denials (18, 11 and 0 after author review).