acceptodds
Under review as a conference paper at ICLR 2027

AGENT MEMORY SCORERS ARE UNRELIABLE ON DENIAL OUTPUTS

Abstract

Agent memory is important for enabling agents to retain, retrieve, and effectively use information from long-term interactions. An LLM agent's memory capability is typically evaluated via *questions*: a fixed *reader* answers each question based on the *memory context* returned by the system under test, and a *scorer* evaluates the answer alone. This work claims that this scoring mechanism is severely biased on questions about an item the recording never contained, whose credited answer is a denial, *no such thing exists*: a claim about the whole recording that no scorer checks. Such questions make up about a quarter of LongMemEval-V2 and LoCoMo. On LongMemEval-V2, adding a simple note, *the recording is complete, so what is not listed does not exist*, to the memory context of the released retrieval baseline raises its denial accuracy from 16% to 72% without changing the retrieval result. The same note makes it deny 55 objects that do exist, up from 1, and the score never notices: a false denial and an honest abstention both score **0**. We then propose three specific mitigation strategies, one for each main component of the agent memory evaluation pipeline: *questions*, *reader* and *scorer*. Extensive experiments show that once the scorer checks each denial against the recording, at most 4 of the baseline's 92 credited denials stand. Read off the answer, a denial score measures who asserts completeness; read against the recording, who can show it.

Primary area:
datasets and benchmarks
Keywords:
agent memory, LLM evaluation, benchmark validity, unanswerable questions, false premise, closed-world reasoning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_RHJXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper examines how agent memory benchmarks, mainly LongMemEval-V2 and LoCoMo, score questions about things that never happened. In these benchmarks, a memory system retrieves information from a long interaction history, a fixed language model answers questions using that information, and a scorer grades only the final answer. About a quarter of the questions contain a false premise (e.g., asking which button appears between two buttons that are actually adjacent), and the correct answer is to say that no such thing exists.

The authors argue that this scoring setup rewards systems for confidently saying “that doesn’t exist,” whether or not they actually checked. Two features of the scorer combine to cause this:

The scorer only checks whether the answer matches the correct one. It never looks at what information the system retrieved. So an answer of “that doesn’t exist” gets full credit even if the system never saw the relevant part of the history and was simply guessing. Guessing is never penalized. Answering “I don’t know” scores 0, and so does a wrong answer. A system that is unsure therefore loses nothing by guessing “that doesn’t exist”: on a false-premise question the guess is right, and on a normal question it scores the same 0 that “I don’t know” would have.

Together, these mean the best strategy is to answer “that doesn’t exist” whenever the system is unsure, regardless of what it actually remembers.

To show this, they add a single paragraph to the retrieved context, stating that the history is complete and anything not listed doesn't exist. This raises accuracy on these questions from 16% to 72%, and even a system with no memory at all reaches 57%. This introduces a new failure which the score doesn't directly show: the system now wrongly denies the existence of things that do exist far more often (55 times instead of once).

The authors propose three fixes. They are:

  • pairing each false-premise question with a matched normal question;
  • standardizing the instructions given to the answering model;
  • a scorer that checks each denial against the actual interaction history.
Critical strengths

S1. Thorough and well-controlled experiments. The central claim, that the benchmark's false-premise questions can be scored highly without using memory, is supported by several independent lines of evidence:

  • the same fixed answer given to every question, with no memory and no model reading anything, scored under four judge models from four vendors (Table 1);
  • a comparison of no memory vs. real retrieval, each with and without the added paragraph (§3.2);
  • a breakdown of which part of the added paragraph earns the credit (Table 3).

The appendix anticipates the usual objections to a critique paper and performs several ablations which strengthen the overall claims.

Critical weaknesses

W1. Limited novelty and undemonstrated practical impact. The core mechanism is a known one, and the authors identify it themselves: Kalai et al. (2025) showed that when a wrong answer and an "I don't know" score the same, guessing is always the better strategy (L092–093). This paper applies that observation to one memory benchmark, where the guess is "this doesn't exist."

The paper's most novel component, a check of whether each "doesn't exist" answer was backed by evidence in the interaction history, is by the authors' own account a prototype. It covers only 92 of the 128 relevant questions and cannot be applied to the second benchmark (L483–484).

The paper's second benchmark (LoCoMo) adds little. Its false-premise questions were already known to be unusable: the benchmark's own evaluation code crashes on 444 of these 446 questions (L2300–2301), and prior audits note that these questions are usually left out of reported scores (L2286–2288). Showing that a part of a benchmark that is already broken and already ignored can also be gamed tells us little that is new.

Most importantly, the paper does not show that the problem affects any published result. The authors state that they "do not claim that published rankings are inflated" (L471). The one reference system they reran gained nothing detectable from its built-in hint, and the top-ranked system was not tested. The paper therefore shows that the score can be inflated, but not that this changes any conclusion drawn from the benchmark.

W2. The writing makes a simple finding hard to understand and verify. The paper's core idea is simple: the benchmark gives credit for answering "this doesn't exist" without checking whether the system had any evidence for that answer. Yet the paper is very difficult to follow. It introduces many paper-specific terms and reports several numbers and confidence intervals in nearly every sentence, so the main argument is buried under detail. Even with a background in this area, I needed multiple careful passes to reconstruct how answers are scored and what each reported percentage measures.

The main text also does not stand on its own. It contains over 40 references to the appendix, and understanding several central claims, including how the paper's main proposed fix works, requires reading the appendix. This matters because a critique paper's value depends on readers being able to check its claims, and the current presentation makes that unusually hard.

I would encourage the authors to build the main text around one worked example, define each term before using it, and keep only the numbers essential to the argument in the main paper.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to RHJX (1/2): W1d, the false-premise questions already decide published comparisons

Thank you for the careful review, and for finding the experiments "thorough and well-controlled" and the central claim "supported by several independent lines of evidence". We label the four parts of Weakness 1 in the order of your review: W1a (Kalai et al.), W1b (the check as a prototype), W1c (LoCoMo) and W1d (published results). The note is our fixed text in the memory context stating that the recording is complete; the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:

  • W1d (the point you mark as most important). The false-premise questions already decide published comparisons, and LongMemEval-V2's top reference system earns its premise-awareness gains through the channel we study. The revision replaces the disclaimer at line 471 with this evidence (§6) and keeps its narrow form in the Limitations: we did not rerun leaderboard systems, and we claim no published conclusion is wrong.
  • W1a. We agree that the incentive is the one Kalai et al. identify. The core difference: Kalai et al. concern wrong answers that binary grading fails to penalise; here the guessed denial is correct, and what fails is the attribution of its credit to memory.
  • W1b. None of the twelve published treatments of false-premise questions we audited reads what the reader was given. The check covers 92 questions because of our page annotation; for an estimated 19 of 128, no list can settle the denial.
  • W1c. We agree that the defect is known; four of the ten published LoCoMo treatments we audited still count these questions, and in REMem's and MAGMA's tables they decide published orderings.
  • W2. We agree, and the revision follows your suggestion: one worked example, terms defined before use, and 15 appendix pointers in the main text instead of 45.

W1d. "The paper does not show that the problem affects any published result." The weakness is visible in published results.

Removing LoCoMo's 446 false-premise questions reverses two published orderings in REMem's tables (ICLR 2026) and one in MAGMA's (ACL 2026), recomputed from the published per-category scores (which reproduce each overall score to within 0.1):

Published comparisonAll 1,986 questionsWithout the 446 false-premise questions
REMem-S vs. full context (REMem)77.5 vs. 76.781.1 vs. 83.0
Graphiti vs. Mem0 (REMem)52.5 vs. 49.743.7 vs. 50.5
A-MEM vs. MemoryOS (MAGMA)58.0 vs. 55.357.0 vs. 58.9

On the false-premise column, Graphiti ranks first of ten methods (83.0), above the oracle handed the gold evidence (70.6), while refusing 954 of 1,986 questions; by REMem's own refusal precision it is last of the three systems REMem measures (38.9%, against 73.3% for REMem and 40.0% for Mem0). We find no evidence that any system was built to game the score; the point is that the column deciding these comparisons cannot tell a grounded denial from an asserted one.

On LongMemEval-V2, the top reference system earns its premise-awareness gains through the channel we study. AgentRunbook-C is instructed to "hint at the downstream reader" in the memory context when the evidence contradicts the premise, and AgentRunbook-R's reranker to tell the downstream model to "call out the flawed premise"; the benchmark paper attributes AgentRunbook-C's abstention gains to this hint. The scorer cannot check "the evidence contradicts the premise", so it pays for the claim either way:

  • As you note, AgentRunbook-R's shipped hint gains nothing detectable: issued on only 14 of 128 questions, it is worth +1.6 (not significant). In the same slot of its own rerank stage, our unconditional note is worth +46.1 points (both against the reranker with the call-out line removed), and false denials rise from 3 to 69. AgentRunbook-C needs a paid coding-agent binary and was not rerun.
  • Under the official reader and the official 200K-token budget, the note raises the released baseline's denial score from 16.4 to 73.4 (73 questions gained, none lost) and its overall score by 19.3 points, with retrieval unchanged.

In our rerun, the corrections demote the credit that answer text buys. Under denial precision and pair-conditioning, a no-memory system with the note falls from second to last of five, and the two real memory systems keep their order. Only the recording check passes a known-answer test: on a complete right-page list, the same list with half its labels removed and a wrong-page list, which the denial score cannot separate (75.0, 70.3 and 71.1; not significant), it accepts 29, 17 and 0 of the credited denials (18, 11 and 0 after author review).

AuthorsOpenReview ↗

Response to RHJX (2/2): W1a–W1c, an error penalty cannot remove a correct guessed denial, only the recording check grounds it, and LoCoMo still counts it; W2, the rewrite

W1a. "The core mechanism is a known one." We agree that the incentive is the one Kalai et al. identify, and the paper cites them for it (lines 092–093). The core difference is what goes wrong: Kalai et al. concern wrong answers that binary grading fails to penalise; on a false-premise question the guessed denial is correct, so an error penalty cannot remove its credit. Rescoring all 451 questions under Kalai et al.'s penalty of t/(1−t) per wrong answer:

Baseline, without and with the noteCorrectWrongNote's gain at a penalty of 1 / 3 / 9
All 451 questions155 → 233208 → 204+82 / +90 / +114 (net score)

The penalty does charge the note on the answerable questions, where wrong answers rise from 139 to 170 and false denials from 1 to 55; on the false-premise questions it cannot, because there the note's denials are correct: correct answers rise from 20 to 92, while wrong answers fall from 69 to 34 and UNKNOWNs from 39 to 2. What is missing from those denials is evidence, which no error count sees; of the 92 credited denials, the recording check accepts 4 (71 of the 92 fall on questions it covers).

The setting also differs from the one Kalai et al. study. In Kalai et al., the model guesses on its own, so a confidence target stated in its instructions reaches the guesser. Here the system under test writes the reader's input (lines 042–046), and the note supplies exactly the confidence a target asks for: a reader told to distrust completeness claims in the memory context still credits the note with +48 points (Table 4).

W1b. "The paper's most novel component ... is a prototype."

  • Novelty. Of the twelve published treatments of false-premise questions we audited, each either drops these questions or scores the answer alone; none reads what the reader was given (App. H). A positive answer can be traced to one passage; a denial needs evidence that the reader saw everything the recording holds on the point, which is what the check tests, and it passes the known-answer test of W1d.
  • Coverage. LongMemEval-V2 does not publish each question's page, so we annotated pages and confirmed 92. 30 of the other 36 are settleable in principle: 5 lack a confirmed page in our annotation, and 25 compare two forms or span several pages, which needs the benchmark's page records and a two-page check we have not implemented. For an estimated 19 of 128 (95% interval 12–28; App. J), no list can settle the denial; we score those by pair-conditioning and denial precision.
  • Second benchmark. On LoCoMo, requiring the annotated evidence session removes all credit of a no-memory system (100 to 0) but leaves BM25 at 86.8 of 87.9: necessary, not sufficient. The full check needs a per-question scope list only the benchmark can write; we have not run it.

W1c. "The paper's second benchmark (LoCoMo) adds little." We agree that the defect is known; what the paper adds is the mechanism, and published systems still count these questions. The crash at lines 2300–2301 concerns only the official forced-choice construction. Four of the ten published LoCoMo treatments we audited count the false-premise questions in their headline score or ranking (A-Mem, MAGMA, REMem, HiGMem; App. H), and in REMem's and MAGMA's tables they decide three orderings (W1d). On LoCoMo the answer instruction, not memory, sets the score: in our end-to-end rerun, Mem0 scores 42.6 with no instruction, 0.0 with the prompt of its own evaluation code, and 96.9 with one instruction shared by all systems; released scorers order the same pair of conditions oppositely in 46–55% of the combinations we replayed (App. M.2).

W2. "The writing makes a simple finding hard to understand and verify." We agree, and the revision follows your suggestion. §1 is built around one worked example, drawn in Figure 1a:

The question asks which action lies between "Reorder" and "Ship" on the toolbar of an order page; the gold is a denial, because the two are adjacent. The released baseline retrieves a different page, so the reader answers UNKNOWN and scores 0. With the note added to the same memory context, the reader denies and is credited, although the toolbar never reached it. The recording check finds no complete toolbar list in the memory context and does not accept the denial.

Terms are defined in a short list at the start of §2, and coined terms are replaced (false-premise question for denial question, recording check for licence, source question for twin). §4.3 explains the recording check in Algorithm 1, and the main text points to the appendix 15 times instead of 45. Changes are marked in blue.

Thank you again for your careful review. If these answers address your concerns, we would be grateful if you would reconsider your assessment.

I thank the authors for their rebuttal. Prior to the rebuttal, my soundness score of the paper was “3: I have no substantive concerns about the paper’s soundness,” with no questions to authors and it remains the same.

The paper is a critique paper of a popular benchmark. It’s rare to have such papers, and it’s difficult to evaluate their contributions and recommend a rating.

Critique papers are useful and should be encouraged in AI research; they show us common pitfalls to avoid. Making benchmarks is common, but making a good benchmark is hard, and sometimes entire research directions are built on the foundations of a benchmark (ARC-AGI being a prominent example).

This paper focuses on LongMemEval-V2, which is also quite a popular benchmark. Agent memory has become a big subfield, with many startups working to improve agents’ memory systems and millions of dollars being spent on it. LongMemEval-V2 is a central part of this subfield.

I don’t have any substantial concerns. The writing in the original submission was unclear, but the revised sections I checked have improved. In light of the rebuttal, I am raising my score from 2 to 3.

At the same time, it’s hard for me to get excited about the work, which is what keeps me from a higher score. The scope of the paper is very narrow: authors have shown 3 examples of wherein the memory system ranking has been changed because of the issue they find in the benchmark. But those are point estimates and the gap seems quite small too. Rerunning them with multiple model backends and providing confidence intervals might not show the ranking ordering to be swapped.

Since critique papers are difficult to assess with standard criteria, and the reviewer scores cluster around borderline without substantive technical objections, I recommend that the AC look at the submission holistically rather than focusing entirely on our scores, and make an appropriate decision based on their experience in research and reviewing.

I am happy to answer any questions that the authors or AC may have.

AuthorsOpenReview ↗

Thank you for your guidance

Dear Reviewer RHJX,

Thank you very much for reading our rebuttal and for your updated assessment. We are sincerely grateful for the time and care you have devoted to our paper. Your guidance on presentation, in particular your suggestion to build the main text around one worked example and to define each term before use, has made the paper much clearer. Your comment on whether the problem affects published results also guided us to connect our findings to published work, which has helped strengthen the practical relevance of the paper.

We are also grateful for your thoughtful remarks on critique papers and on the role of benchmarks in shaping a research direction. We sincerely hope our work can contribute to the agent memory community by helping build evaluations that reward what a memory system actually remembers.

Thank you once again for your generous and valuable guidance.

Best regards, The Authors

Reviewer_5EoFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

On memory benchmarks like LongMemEval-V2 and LoCoMo, about a quarter of the questions ask about things that never happened, and the credited answer is a denial. The scorer reads only the answer, so it can't tell whether a denial is grounded in memory or just asserted.

Adding a single note to the memory context ("the recording is complete, so what is not listed does not exist") raises denial accuracy from 16% to 72% with no better retrieval. It also makes the agent deny many things that do exist, which the score never counts.

The paper proposes three fixes. The best one checks each denial against the actual recording and credits it only if the context the system delivered really listed everything relevant.

Critical strengths
  1. It exposes a serious flaw in widely used memory benchmarks and backs it with extensive evidence.

  2. It proposes a check that separates grounded denials from asserted ones.

Critical weaknesses
  1. The prose is extremely dense, with many coined terms and paragraphs packed with numbers and appendix pointers. The main argument is spread across 47 pages.

  2. The recording check works only where a page's contents can be listed exhaustively, which excludes about a third of the denial questions. It depends on a page mapping that the benchmark hasn't published and the authors had to reconstruct.

Questions for authors
  1. Do any published rankings actually change? If denial precision or the recording check were applied to submitted systems, would any system's rank move?

  2. Can benchmark users run the recording check? It relies on a question-to-page mapping that LongMemEval-V2 doesn't publish. Is there a version outside users could run, or does it have to be done by the organizers?

  3. What should happen to denial questions that can't be checked? About a third have no single page whose contents can be listed exhaustively.

2 comments
AuthorsOpenReview ↗

Response to 5EoF (1/2): do rankings change?

Thank you for the clear summary and the constructive questions, and for finding that the paper "exposes a serious flaw in widely used memory benchmarks and backs it with extensive evidence". Below, the note is our fixed text in the memory context stating that the recording is complete, and the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:

  • Question 1 (do rankings change?). Published orderings already depend on the false-premise questions, and in our rerun the corrections drop a no-memory system with our note from second to last while keeping the order of the two real memory systems. We could not rescore the LongMemEval-V2 leaderboard itself: submissions go to the organisers and are not public.
  • Weakness 2, Question 2 (can outside users run the check?). Yes, on their own runs, with one command; the organisers can run it on every submission.
  • Weakness 2, Question 3 (questions that cannot be checked). The check covers 92 of 128 questions today; for an estimated 19, no list can settle the denial, and we score those separately.
  • Weakness 1 (density). We agree. The revision is rebuilt around one worked example and points to the appendix 15 times in the main text instead of 45.

Question 1. Published results: the false-premise questions decide three published orderings. Recomputed from the published per-category scores, which reproduce each overall score to within 0.1:

Published comparisonAll 1,986 questionsWithout LoCoMo's 446 false-premise questions
REMem-S vs. full context (REMem, ICLR 2026)77.5 vs. 76.781.1 vs. 83.0
Graphiti vs. Mem0 (REMem)52.5 vs. 49.743.7 vs. 50.5
A-MEM vs. MemoryOS (MAGMA, ACL 2026)58.0 vs. 55.357.0 vs. 58.9

On the false-premise column, Graphiti ranks first of ten methods (83.0), above the oracle handed the gold evidence (70.6), while refusing 954 of 1,986 questions; under REMem's own refusal precision, the analogue of our denial precision, Graphiti is last of the three systems REMem measures (38.9%, against 73.3% for REMem and 40.0% for Mem0). We do not claim that any system was built to game the score or that these papers' conclusions are wrong; the column that decides them cannot tell a grounded denial from an asserted one.

LongMemEval-V2: the top reference system earns its premise-awareness gains through the channel we study. AgentRunbook-C is instructed to "hint at the downstream reader" in the memory context when the evidence contradicts the premise, and AgentRunbook-R's reranker to tell the downstream model to "call out the flawed premise"; the benchmark paper attributes AgentRunbook-C's abstention gains to this hint. The scorer cannot check "the evidence contradicts the premise", so it pays for the claim either way:

  • Inside AgentRunbook-R's own rerank stage, our note in the slot of its hint is worth +46.1 points against the reranker with that line removed, and false denials rise from 3 to 69. Its shipped conditional hint (14 of 128 questions) is worth +1.6, not significant.
  • Under the official reader and the official 200K-token budget, the note raises the released baseline's denial score from 16.4 to 73.4 (73 questions gained, none lost) and its overall score by 19.3 points, with retrieval unchanged.

Our rerun on LongMemEval-V2 (128 false-premise questions):

SystemDenial scoreDenial precisionPair-conditioned
Baseline + note71.9 (1)63% (3)39.8 (1)
No-memory system + note57.0 (2)38% (5)4.7 (5)
AgentRunbook-R + note56.2 (3)51% (4)25.8 (2)
Baseline15.6 (4)95% (1)10.2 (3)
AgentRunbook-R11.7 (5)75% (2)7.0 (4)

Both corrections drop the no-memory system from second to last and keep the baseline ahead of AgentRunbook-R, the order of their answerable accuracy (41.8 vs. 35.3); only the recording check also removes the credit the note buys the baseline (1.6 to 3.1 instead of 15.6 to 71.9). On lists we build ourselves, the denial score cannot tell a complete right-page list, the same list with half its labels removed and a wrong-page list apart (75.0, 70.3, 71.1; not significant), while the recording check accepts 29, 17 and 0 of their credited denials (18, 11 and 0 after author review).

AuthorsOpenReview ↗

Response to 5EoF (2/2): running the check, questions that cannot be checked, and the writing

Weakness 2 and Question 2. Outside users can run the recording check on their own runs today, and the organisers on every submission. The official harness stores the memory context the reader received, after truncation, in each per_question.jsonl, and every submission must include that file; the script needs nothing else from a submission and calls no model:

python3 rescore_submission.py <submission folder>

It prints the denial score, the pair-conditioned score and the recording check; denial precision also needs false-denial labels (--false-denials) or a judge (--judge-model).

  • Already in the submitted supplement: our author-confirmed question-to-page mapping (data/anchor_human.json) and the source-question map (data/twin_map.json).
  • Added in the revised supplement (rebuttal_analysis/ and runs/think_prompts/): the script; the recorded labels of every element type on each of the 92 confirmed pages (data/anchor_families.json); the reader inputs of our runs; and the author's false-denial labels of the baseline pair. From these the script reproduces the paper's numbers (for the baseline with the note: 71.9, 63% precision, 39.8 and 4 grounded denials).

Three ways to obtain the page mapping:

  • Our mapping: 92 of the 128 questions have an author-confirmed page.
  • Automatic mappings from public data alone: they still separate a right-page list from a wrong-page list (40.2% and 40.8% of credited denials accepted, against 13.8% and 15.2%), but only 34–43% of the accepted denials sit on the right page, so they can compare two systems, not report an absolute rate.
  • The benchmark's own mapping: the organisers hold it and can pass it to the script (--data-dir).

Weakness 2 and Question 3. Questions the check cannot settle: an estimated 19 of 128, scored separately.

Our page annotationQuestionsSettleable by an exhaustive check (App. J)
Single confirmed page9279
No single page (two forms compared, multi-page workflows)3125
Left open55
All128109

30 of the 36 questions outside the current check are settleable in principle: 5 lack a confirmed page in our annotation, and 25 compare two forms or span several pages, which needs the benchmark's page records and a two-page check we have not implemented. Most of the 19 that no list can settle ask about workflow steps or site capabilities rather than items on a page. The figure is an estimate (95% interval 12–28): model labels from two families, with every disagreement and a sample of 30 agreements adjudicated by an author. The revised §4.3 reports these questions by pair-conditioning and denial precision, separately from the ranked score; a comparison of two pages would need both pages' lists, which we have not implemented.

Weakness 1. We agree that the prose was too dense. The revision builds §1 around one worked example (Figure 1a), explains the recording check in an algorithm box (Algorithm 1, §4.3), points to the appendix 15 times in the main text instead of 45, and replaces coined terms, defining the rest in a short list at the start of §2:

Submitted PDFRevised PDF
Denial question / sound-premise questionFalse-premise question / answerable question
Denial accuracy, published scoreDenial score
Store, empty storeMemory system, no-memory system
TwinSource question
Licensed denial, licenceGrounded denial, recording check
Inventory, family, anchorList, element type, page mapping
Directive, answer policyAnswer instruction

Changes are marked in blue. We hope these answers address your three questions and the soundness concerns behind them; if so, we would be grateful if you would consider raising your score.

Reviewer_hF2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the reliability of existing agent memory benchmarks, focusing on questions that require denying the existence of an item. The authors show that a simple instruction can significantly improve evaluation scores without improving memory retrieval. They further analyze this issue on LongMemEval-V2 and LoCoMo and propose several changes to the evaluation process.

Critical strengths
  • The paper identifies an interesting issue in existing agent memory benchmarks. The experiments show that evaluation scores can be improved by changing the answer policy rather than improving memory retrieval. This is a useful observation for future benchmark design.
Critical weaknesses
  • The paper mainly analyzes scoring issues in existing benchmarks and proposes modifications to their evaluation protocols. While the findings are useful, the paper does not introduce a new benchmark or a new agent memory method. The overall contribution feels somewhat incremental. I think this work may be more suitable for an NLP evaluation venue, such as ACL, than ICLR.

  • The proposed evidence-based scoring method relies on access to the original records and structured page information. It is unclear how well this method would work for more general agent memory tasks, especially those involving open-ended conversations or less structured information. The paper would be stronger with validation beyond the current settings.

Questions for authors
  1. How can the proposed scoring method be extended to more general agent memory tasks, such as open-ended conversations or multimodal memory tasks?

  2. Beyond identifying weaknesses in existing benchmarks, what is the main methodological contribution of this work compared with prior studies on evaluation reliability?

3 comments
AuthorsOpenReview ↗

Response to hF2D (1/2): the methodological contribution, and why we repair the scorer

Thank you for the constructive review, and for finding the issue interesting and the finding "a useful observation for future benchmark design". Below, the note is our fixed text in the memory context stating that the recording is complete, and the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:

  • Question 2 (methodological contribution). A scorer that credits a denial only when the recording supports it, an audit that measures how much of a score a fixed answer text buys, and mitigations that a leaderboard can adopt without rerunning any system.
  • Weakness 1 (relevance). The false-premise questions already decide published comparisons: removing them reverses two published orderings in REMem's LoCoMo tables (ICLR 2026) and one in MAGMA's (ACL 2026).
  • Weakness 1 (venue). The paper's primary area, datasets and benchmarks, is an ICLR subject area, and the closest precedent was an oral at ICLR 2025.
  • Weakness 2, Question 1 (generality). The audit and two of the three mitigations need only the released scorer; the recording check needs a list of what the recording holds on each question's point, which a benchmark can also write for conversation.

Question 2. The methodological contribution is a way to measure, and to remove, the credit that an answer-only scorer gives to denials without evidence.

  • Recording check (§4.3). It credits a correct denial only if the memory context lists every recorded item of the queried kind on the question's page and the queried item is not in the list. On lists whose evidence we fix in advance, the denial score cannot separate a complete right-page list, the same list with half its labels removed, and a complete wrong-page list (75.0, 70.3 and 71.1; not significant); the recording check accepts 29, 17 and 0 of their credited denials (18, 11 and 0 after author review).
  • Audit (§3). We hold retrieval fixed and change only the text the system under test writes for the reader. A fixed answer written without memory is credited on three benchmarks (LongMemEval-V2, LoCoMo, BEAM) under both kinds of scorer, an LLM judge and a string match, and our note's gain holds for three reader models.
  • Mitigations (§4). One answer instruction shared by all systems shrinks the gap between three LoCoMo memory systems from 31 points to 9; pair-conditioning and denial precision drop a no-memory system with the note from second to last on LongMemEval-V2, while the two real memory systems keep their order.

Prior work on evaluation reliability looks elsewhere. Null-model studies change the answer a system outputs (Zheng et al., 2025); Kalai et al. (2025) explain why a model guesses; REMem reports refusal precision on the answers. Of the twelve published treatments of false-premise questions that we audited, each either drops these questions or scores the answer alone; none reads what the reader was given (App. H).

Weakness 1 (contribution). We repair the scorer rather than build a new benchmark, because the weakness sits in how denials are scored, and a repaired scorer can rescore past results. The recordings and most questions of LongMemEval-V2 and LoCoMo are carefully built and widely used; the weakness appears when the system under test writes the reader's input and the scorer reads only the answer. Newer memory benchmarks that add false-premise probes still score the answer alone (§5), so a new benchmark would inherit the weakness and could not rescore existing results. The supplement includes a script that rescores any LongMemEval-V2 submission.

Weakness 1 (relevance). Published comparisons already depend on these questions. Recomputed from the published per-category scores:

Published comparisonAll 1,986 questionsWithout LoCoMo's 446 false-premise questions
REMem-S vs. full context (REMem)77.5 vs. 76.781.1 vs. 83.0
Graphiti vs. Mem0 (REMem)52.5 vs. 49.743.7 vs. 50.5
A-MEM vs. MemoryOS (MAGMA)58.0 vs. 55.357.0 vs. 58.9

On LongMemEval-V2, the benchmark paper attributes its top reference system's abstention gains to a premise hint that the system writes into the memory context, a claim the scorer cannot check.

AuthorsOpenReview ↗

Response to hF2D (2/2): venue, open-ended conversation and other forms of memory

Weakness 1 (venue). We respectfully believe that the paper fits ICLR.

  • ICLR's reviewer guidelines value submissions that convincingly demonstrate new, relevant and impactful knowledge, including empirical knowledge, and ICLR's call for papers lists datasets and benchmarks among its topics; it is the paper's primary area.
  • The closest precedent, Zheng et al.'s study showing that a constant response tops judge-based leaderboards, proposed no new benchmark or model and was an oral at ICLR 2025.
  • The paper studies the evaluation of learned agent systems: LongMemEval-V2 records web-agent trajectories, and several of the benchmarks and systems we audit appeared at ICLR and NeurIPS (LongMemEval, BEAM, LightMem, REMem, A-Mem).

Weakness 2 and Question 1. Most of the method needs no structured pages; the recording check needs a list of what the recording contains, which a benchmark can supply for conversation too.

1. The audit transfers unchanged, and so do the two mitigations that need only the scorer.

LongMemEval-V2LoCoMoLongMemEval-S
What is recordedWeb and enterprise agent trajectoriesOpen-ended conversations between two peopleChat histories between a user and an assistant
ScorerLLM judgeString matchLLM judge
Audit resultA fixed denial written without memory is credited on 93–100% under four judge modelsA fixed "not mentioned" written without memory scores 100%The note over BM25 retrieval raises abstention accuracy from 66.7 to 100
MitigationPair-conditioning: source question found for all 128Shared answer instruction: gap between three memory systems from 31 points to 9Pair-conditioning applicable: 29 of 30 abstention questions share an ID with their source question

2. The benchmark runs the recording check, and always holds the original recording. The system under test supplies nothing extra; page renderings are how LongMemEval-V2 supplies the list, not a requirement of the method.

3. In open-ended conversation, delivering the evidence is necessary but not sufficient. LoCoMo annotates the evidence turn of all 446 false-premise questions. Keeping a credited denial only if its evidence session reached the reader:

LoCoMo, reader told to answer "not mentioned" when the context is silentDenial scoreWith the evidence requirement
No-memory system1000
BM25 (delivers the evidence session on 441 of 446)87.986.8

The requirement removes every credit of the no-memory system; a grounded denial still needs a complete list. A benchmark holds the full transcript and can list, for each question, everything the conversation records about the question's person, topic and time span; the recording check then asks whether the memory context covers the list, as for a page. We have not run this full check, because only the benchmark can write the lists; the revised §6 states the protocol.

4. Multimodal memory needs the same thing: a listable description of each recorded state. LongMemEval-V2 is itself multimodal: each recorded state has a screenshot and an accessibility tree, and the recording check reads the interface labels in the accessibility tree. A multimodal benchmark that records listable descriptions, such as interface trees or object annotations, supports the same check; where a recording has none, the shared answer instruction and pair-conditioning remain available.

We hope these answers address your concerns about the contribution and the generality of the method. If so, we would be grateful if you would consider raising your score; we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

A gentle follow-up before the discussion period closes

Dear Reviewer hF2D,

Thank you again for your constructive review and for the time you have devoted to our paper. As the discussion period is drawing to a close, we would like to briefly follow up and check whether our response has addressed your concerns. In brief:

  1. Contribution. As explained in our response, we see the methodological contribution as a scorer that credits a denial only when the recording supports it, an audit that measures how much of a score a fixed answer text buys, and mitigations that a leaderboard can adopt without rerunning any system. We also showed that the issue affects published results: removing the false-premise questions reverses three published orderings in REMem's and MAGMA's LoCoMo tables.
  2. Generality. The audit and two of the three mitigations need only the released scorer, so they also apply to open-ended conversation (LoCoMo) and chat histories (LongMemEval-S). The recording check needs only a list of what the recording holds on each question's point, which a benchmark can also write for conversation; on LoCoMo, requiring the evidence session alone removes all of a no-memory system's credit (100 to 0). Your question on generality also led us to describe this protocol for conversation in the revised §6.

We sincerely hope these answers address your concerns about the contribution and the generality of the method. If so, we would be truly grateful if you could kindly reconsider your score. If any concerns remain, we would be glad to address them before the discussion closes.

Thank you once again for your time and thoughtful feedback.

Best regards, The Authors

Reviewer_vZydOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper identifies a weakness in existing agent-memory benchmarks: roughly one-quarter of the questions require denying the existence of information in the interaction history. Existing evaluations cannot distinguish between denials supported by evidence and unsupported assertions. This allows systems to receive credit without retrieving sufficient evidence, which does not lead to realistic assessments. The authors demonstrate that adding a directive asserting memory completeness increases the accuracy of denials from 15.6% to 71.9%, while also increasing the number of false denials of existing information from 1% to 55%. The authors find that an empty memory system achieves 57% accuracy on denial questions using this directive. Recording-based verification suggests that only four of the 92 credited denials are actually supported, which highlights a validity problem in current benchmark evaluation practices.

Critical strengths
  • An important and clearly demonstrated flaw in existing benchmarks is shown.
  • The authors investigate whether the observed effect depends on specific experimental choices and demonstrate that it persists across many variations.

The paper can serve as a motivation to build better agent-memory benchmarks in the future.

Critical weaknesses
  • The practical consequences for existing leaderboards remain uncertain given that the paper demonstrates that the scoring mechanism can theoretically be manipulated. Is there any evidence that published systems have actually exploited this weakness or that their rankings would change substantially under corrected scoring?

  • The evaluation primarily measures how the three mitigation strategies alter the behavior of existing systems and experimental interventions. But do these strategies produce better memory system rankings? This remains unclear to me...

Questions for authors

The central finding is convincing and highlights a weakness in the evaluation of agent-memory systems. This is a relevant result. The experimental design is effective, and demonstrating that an empty memory system can outperform a real retrieval baseline with a simple completeness assertion is particularly compelling. I have several questions:

  • The main experiments differ from the official LongMemEval-V2 protocol. The authors use a smaller context budget, non-thinking greedy decoding, a shorter completion budget, and a substitute evaluator model. Although additional experiments show that the central effect persists under the official reader configuration and context budget, the results cannot be interpreted as exact reproductions of the official leaderboard. The authors should comment on this.

  • Should denial questions evaluate factual correctness, evidential justification, or both? Is it possible to formalize these objectives and explain how the proposed scoring procedure balances them?

  • How would the proposed verification approach generalize to unstructured conversational memory, where completeness cannot be established through page-level enumeration?

  • Do the proposed mitigations change the relative rankings of existing memory systems? Can you demonstrate that they correlate better with independently assessed memory quality?

2 comments
AuthorsOpenReview ↗

Response to vZyd (1/2): published results, and rankings

Thank you for the careful review, and for finding the flaw "important and clearly demonstrated" and the central finding "convincing". Below, the note is our fixed text in the memory context stating that the recording is complete, and the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:

  • Weakness 1. Published results are already affected: the false-premise questions decide three published orderings on LoCoMo, and LongMemEval-V2's top reference system earns its premise-awareness gains through the channel we study.
  • Weakness 2, Question 4. The corrected scores keep the order of the two real memory systems we reran and drop a no-memory system with the note to last; only the recording check also removes the note's credit and passes a known-answer test.
  • Question 1. Our runs are not a leaderboard reproduction; restoring each deviation leaves the effect.
  • Question 2. Both, in one scoring rule with two weights (revised §4.3).
  • Question 3. The full check needs a scope list only the benchmark can write; we tested a weaker evidence requirement.

Weakness 1. The false-premise questions already decide published comparisons.

Removing LoCoMo's 446 false-premise questions reverses two published orderings in REMem's tables (ICLR 2026) and one in MAGMA's (ACL 2026) (recomputed from the published per-category scores, which reproduce each overall score to within 0.1):

Published comparisonAll 1,986 questionsWithout the 446 false-premise questions
REMem-S vs. full context (REMem)77.5 vs. 76.781.1 vs. 83.0
Graphiti vs. Mem0 (REMem)52.5 vs. 49.743.7 vs. 50.5
A-MEM vs. MemoryOS (MAGMA)58.0 vs. 55.357.0 vs. 58.9

On the false-premise column, Graphiti ranks first of ten methods (83.0), above the oracle handed the gold evidence (70.6), while refusing 954 of 1,986 questions; by REMem's own refusal precision it is last of the three systems REMem measures (38.9%, against 73.3% for REMem and 40.0% for Mem0). We find no evidence that any system was built to game the score, and we do not claim these papers' conclusions are wrong: the comparisons are close, and the column that decides them cannot tell a grounded denial from an asserted one.

On LongMemEval-V2, the top reference system earns its premise-awareness gains through the channel we study. AgentRunbook-C is instructed to "hint at the downstream reader" in the memory context when the evidence contradicts the premise, and AgentRunbook-R's reranker to tell the downstream model to "call out the flawed premise"; the benchmark paper attributes AgentRunbook-C's abstention gains to this hint. The scorer cannot check "the evidence contradicts the premise", so it pays for the claim either way:

  • Inside AgentRunbook-R's own rerank stage, our note in the slot of its hint is worth +46.1 points against the reranker with that line removed, and false denials rise from 3 to 69; its shipped conditional hint (14 of 128 questions) is worth +1.6, not significant.
  • Under the official reader and the official 200K-token budget, the note raises the released baseline's denial score from 16.4 to 73.4 (73 questions gained, none lost) and its overall score by 19.3 points, with retrieval unchanged.

Weakness 2 and Question 4. Rankings and memory quality.

LongMemEval-V2, 128 false-premise questionsDenial scoreDenial precisionPair-conditioned
Baseline + note71.9 (1)63% (3)39.8 (1)
No-memory system + note57.0 (2)38% (5)4.7 (5)
AgentRunbook-R + note56.2 (3)51% (4)25.8 (2)
Baseline15.6 (4)95% (1)10.2 (3)
AgentRunbook-R11.7 (5)75% (2)7.0 (4)

Both corrections drop the no-memory system from second to last and keep the baseline ahead of AgentRunbook-R, the order of their answerable accuracy (41.8 vs. 35.3). Pair-conditioning still leaves the note about +30 points on the baseline; only the recording check removes it. (Precision: baseline pair author-labelled, the rest model-labelled; ranks unchanged under model labels.)

Against memory quality fixed independently of any score, only the recording check passes. We built the evidence ourselves; the same reader, with the note, receives:

Memory contextDenial scoreAccepted by the recording checkAfter author review
Complete list, right page75.02918
Same list, half the labels removed70.31711
Complete list, wrong page71.100

The denial score cannot tell the three apart (not significant); the recording check orders them by the evidence they carry.

AuthorsOpenReview ↗

Response to vZyd (2/2): the official protocol, correctness and justification, conversational memory

Question 4, continued. The denial score tracks memory only while the answer text is fixed: it follows answerable accuracy across the 24 conditions that vary only retrieval (Spearman ρ = 0.49, p = 0.015), but not across all 67 conditions, which also vary the answer instruction (ρ = 0.08).

Question 1. We agree: our runs are paired comparisons in a lower-cost configuration, not a reproduction of the leaderboard, and the revised §1 and §2 say so. Restoring each deviation leaves the effect in place.

We deviated to afford 83 conditions of 451 questions each; the official reader thinks for up to 20,000 tokens over up to 200,000 tokens of memory. The reader model, Qwen3.5-9B, is the official one.

Deviation (official setting)Note's effect on the baseline's denial score, deviation restored
16K memory budget (200K)16.4 to 73.4 at 200K with the official reader (32K: 18.8 to 66.4; 65K: 14.1 to 70.3)
Non-thinking greedy decoding (thinking, temperature 0.6)12.5 to 75.0; two more seeds: 12.5 to 80.5, 15.6 to 74.2
768 completion tokens (20,000)as above; 16,384 instead of 20,000 tokens moves no denial score by more than 2.3
Judge gpt-5.2, unavailable to us (deepseek-v4-pro, official prompts)+58.6 and +59.4 under glm-5.2 and kimi-k3 (+56.2 under deepseek-v4-pro)

At the official reader and budget our rerun scores the baseline 40.1 overall, against the published 51.0 (the remaining gap is the substitute judge and a smaller embedder), so we report only differences within one configuration.

Question 2. Both: correctness and justification fit one scoring rule with two weights (revised §4.3).

Correctness: Corr(q, a) = 1 when answer a meets the benchmark's rubric. Justification: Cert(q, C) = 1 when the memory context C lists every recorded item of one element type on the question's page and the denied object is not among them (Algorithm 1); Cert = 0 where the check cannot run. Over the false-premise questions D and the answerable questions S that name an object, with FD the false denials on S:

Sc,λ=1∣D∣∑q∈DCorr(q,aq) [Cert(q,Cq)+c (1−Cert(q,Cq))]−λ FD∣S∣S_{c,\lambda}=\frac{1}{|D|}\sum_{q\in D}\mathrm{Corr}(q,a_q)\,\big[\mathrm{Cert}(q,C_q)+c\,(1-\mathrm{Cert}(q,C_q))\big]-\lambda\,\frac{\mathrm{FD}}{|S|}

c ∈ [0, 1] is the credit for a correct denial without evidence, and λ ≥ 0 the price of denying something that exists; the published denial score is c = 1, λ = 0.

Systemc = 1, λ = 0 (denial score)c = 0, λ = 0c = 0, λ = 1
Baseline15.61.61.2
Baseline + note71.93.1−15.5
No-memory system + note57.00.0−40.3

The note adds 56 points at c = 1, 1.6 points at c = 0, and costs 16.7 points at λ = 1. To rank memory we recommend c = 0 with denial precision beside it; c = 1 measures task success. A memory context that does deliver a complete list scores 22.7 at c = 0 (the right-page list). Questions on which the check cannot run score 0 at c = 0 for every system and are also reported separately.

Question 3. In conversation, the check needs a list of what the conversation records within each question's scope (person, topic, time span); only the benchmark can write it, and we have not run that full check. Pages are how LongMemEval-V2 supplies the list, not a requirement of the method. We tested the weaker requirement that the annotated evidence reach the reader: LoCoMo annotates the evidence turn of all 446 false-premise questions, and we keep a credited denial only if its evidence session was delivered.

LoCoMo, reader told to answer "not mentioned" when the context is silentDenial scoreWith the evidence requirement
No-memory system1000
BM25 (delivers the evidence session on 441 of 446)87.986.8

Delivering the evidence is necessary but not sufficient: it removes every credit of the no-memory system, but a grounded denial still needs a complete list of the scope. The revised §6 states this protocol.

We hope these answers address your concerns. If so, we would be grateful if you would consider raising your score, and we are happy to discuss any remaining point.

Comments on the paper

3 comments
Area_Chair_s4SxOpenReview ↗

Reminder: Please Read and Discuss the Author Rebutta

Dear Reviewers,

Thank you again for your valuable reviews and discussions. As the author-reviewer discussion period will end on November 18 (AoE), I would kindly ask you to take another look at the authors’ rebuttal if you have not yet done so.

Please make use of this final opportunity to discuss any remaining questions or concerns with the authors, particularly if your current assessment is still borderline or uncertain. I hope the discussion can help clarify the key issues and lead to a fair and clear final assessment.

Thank you very much for your time and contribution.

AC, ICLR 2027

AuthorsOpenReview ↗

General response (1/2): summary, and published overall orderings

Dear AC and reviewers,

Thank you for the careful reviews. All four reviewers find the weakness real: vZyd calls it "an important and clearly demonstrated flaw", 5EoF "a serious flaw in widely used memory benchmarks", hF2D "a useful observation for future benchmark design", and RHJX finds the experiments "thorough and well-controlled". Below, the note is our fixed text in the memory context stating that the recording is complete, and the denial score is the share of false-premise questions (the denial questions of the submitted PDF) that the official scorer credits. In brief:

  • Practical impact (vZyd, 5EoF, RHJX). The false-premise questions already decide published comparisons: removing them reverses two published orderings in REMem's LoCoMo tables (ICLR 2026) and one in MAGMA's (ACL 2026), and LongMemEval-V2's top reference system earns its premise-awareness gains through the channel we study.
  • Rankings (vZyd, 5EoF, RHJX). On the two real memory systems we reran, the corrected scores keep their order; they drop a no-memory system that uses our note from second to last, and only the recording check passes a known-answer test on evidence.
  • Generality (all). The audit and two of the three mitigations need only the released scorer. The recording check needs a list of what the recording holds on each question's point; for conversation, only the benchmark can write it.
  • Presentation (5EoF, RHJX, vZyd). The revision is built around one worked example, replaces coined terms, adds Algorithm 1, and points to the appendix 15 times in the main text instead of 45.
  • Contribution and venue (hF2D, RHJX). An audit that separates the answer text from memory, a scorer that credits a denial only when the recording supports it, and mitigations a leaderboard can adopt without rerunning systems; the closest precedent was an ICLR 2025 oral.

1. Published overall orderings (vZyd, 5EoF, RHJX).

The false-premise questions decide published overall orderings. From the published per-category scores, we recompute each method's accuracy without LoCoMo's 446 false-premise questions:

Published comparisonAll 1,986 questionsWithout the 446 false-premise questions
REMem-S vs. full context (REMem, ICLR 2026)77.5 vs. 76.781.1 vs. 83.0
Graphiti vs. Mem0 (REMem)52.5 vs. 49.743.7 vs. 50.5
A-MEM vs. MemoryOS (MAGMA, ACL 2026)58.0 vs. 55.357.0 vs. 58.9

The column that decides them ranks first a system that refuses almost half of all questions, above the oracle. In REMem's tables, Graphiti scores 83.0 on the false-premise questions, first of ten methods and above the oracle handed the gold evidence (70.6), while refusing 954 of 1,986 questions; by REMem's own refusal precision, Graphiti is last of the three systems REMem measures (38.9%, against 73.3% for REMem and 40.0% for Mem0). We do not claim that any system was built to game the score or that these papers' conclusions are wrong: the comparisons are close, and the column that decides them cannot tell a grounded denial from an asserted one.

2. LongMemEval-V2: the top reference system earns its premise-awareness gains through the channel we study (vZyd, 5EoF, RHJX).

The top reference system, AgentRunbook-C, is instructed to "hint at the downstream reader" in the memory context when the evidence contradicts the premise, and AgentRunbook-R's reranker to tell the downstream model to "call out the flawed premise"; the benchmark paper attributes AgentRunbook-C's abstention gains to this hint. The scorer cannot check the condition "the evidence contradicts the premise", so it pays for the claim whether or not the condition holds:

  • Inside AgentRunbook-R's own rerank stage, our note in the slot of its hint is worth +46.1 points against the reranker with that line removed, while its false denials rise from 3 to 69. Its shipped conditional hint, issued on 14 of 128 questions, is worth +1.6 (not significant), so we do not claim that this hint inflates its score.
  • Under the official reader and the official 200K-token memory budget, the note raises the released baseline's denial score from 16.4 to 73.4 (73 questions gained, none lost) and its overall score by 19.3 points, without changing retrieval.
AuthorsOpenReview ↗

General response (2/2): rankings, generality, presentation, contribution and revisions

3. Rankings in our rerun (vZyd, 5EoF, RHJX).

LongMemEval-V2, 128 false-premise questionsDenial scoreDenial precisionPair-conditioned
Baseline + note71.9 (1)63% (3)39.8 (1)
No-memory system + note57.0 (2)38% (5)4.7 (5)
AgentRunbook-R + note56.2 (3)51% (4)25.8 (2)
Baseline15.6 (4)95% (1)10.2 (3)
AgentRunbook-R11.7 (5)75% (2)7.0 (4)

Denial precision: the share of a system's denials that are correct. Pair-conditioned: a denial counts only if the same system answers the answerable question it was written from. Both corrections drop the no-memory system from second to last and keep the baseline ahead of AgentRunbook-R, the order of their answerable accuracy (41.8 vs. 35.3). Pair-conditioning still leaves the note about +30 points on the baseline; only the recording check removes that credit. Precision of the baseline pair is author-labelled, the rest model-labelled; with model labels throughout, every rank is unchanged.

Only the recording check passes a known-answer test. We fix the evidence ourselves and give the same reader, with the note, a complete list of the question's page, the same list with half its labels removed, or a complete list of a wrong page. The denial score cannot tell the three apart (75.0, 70.3 and 71.1; not significant); the recording check accepts 29, 17 and 0 of their credited denials (18, 11 and 0 after author review). Of the 92 credited denials of the baseline with the note, 71 fall on questions the check covers, and the check accepts 4.

4. Generality (all reviewers). The audit, the shared answer instruction and pair-conditioning need only the released scorer (pair-conditioning also uses the source-question map we release). We ran the audit on LongMemEval-V2, LoCoMo and LongMemEval-S, pair-conditioning on LongMemEval-V2 and the shared instruction on LoCoMo. In conversation, delivering the annotated evidence is necessary but not sufficient: on LoCoMo, requiring the evidence session removes all credit of a no-memory system (100 to 0) but leaves BM25 at 86.8 of 87.9. The full check needs, for each question, a list of what the conversation records within its scope (person, topic, time span), which only the benchmark can write; we have not run it, and §6 states the protocol.

5. Presentation (5EoF, RHJX, vZyd). The revision builds §1 around one worked example (Figure 1a), defines every term in a short list at the start of §2, replaces coined terms (false-premise question for denial question, recording check for licence), explains the recording check in Algorithm 1 (§4.3), and points to the appendix 15 times in the main text instead of 45.

6. Contribution and Kalai et al. (hF2D, RHJX). Kalai et al. (2025) concern wrong answers that binary grading fails to penalise; here the guessed denial is correct, and what fails is the attribution of its credit to memory. Rescoring under their error penalty, the note still gains at every penalty (+82, +90 and +114 in net score over the 451 questions, at 1, 3 and 9 points per wrong answer): on the answerable questions its wrong answers rise from 139 to 170, but on the false-premise questions its credited denials rise from 20 to 92 while wrong answers fall from 69 to 34. Only reading the delivered context against the recording separates a grounded denial from an asserted one.

7. Venue (hF2D). The paper's primary area, datasets and benchmarks, is an ICLR subject area; the closest precedent, Zheng et al.'s study showing that a constant response tops judge-based leaderboards, proposed no new benchmark or model and was an oral at ICLR 2025.

Revisions in the uploaded PDF (marked in blue): the worked example and the official-configuration result (§1, Figure 1); the term list and the deviations from the official protocol (§2); the ranking tables, Algorithm 1 and the scoring rule (§4.3, and §4.4, which is new); the published results, the reference-system hints, the rescoring of submissions and the check on conversation (§6); the analyses behind them (new App. N); and rebuttal_analysis/ in the supplement. Line and table numbers in our replies refer to the submitted PDF; section numbers and appendix letters are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.