- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that memories shouldn't be judged only on the questions they were written for. In a controlled reward-summing task, cued memories get 100% on expected queries but drop to about 50% when the query shifts, while uncued memories stay at 100%. A larger task and a retrieval audit show that other errors come from how models encode and read memory, or from incomplete retrieved evidence.
Critical strengths
-
The main comparison is cleanly controlled. History, decision rule, and budget are all fixed, and only the writing instruction changes. Two memories with identical in-task scores end up very different on shifted queries, and the uncued memories don't use more space. This makes a clear case that in-task accuracy alone can rank quite different memories as equal.
-
The two-reference scoring is a useful idea. By checking each answer against both the full history and the events the reader actually saw, the paper separates wrong evidence from wrong reading. This could be reused when evaluating memory or RAG systems more broadly.
Critical weaknesses
-
Most findings follow directly from the setup. The cued writer is told every later query will be about one feature, so it reasonably stores only that feature. The programmatic cued cache in Table 2 gets exactly the same 50% on shifted queries, so the LLM result adds little beyond what the setup already implies. The retrieval audit is similar. A reader given a subset of events follows the sum of those events, which is again expected.
-
The evidence is narrow, and beyond the small task the results are mixed. The tasks use 6 or 12 synthetic events, with 30 and 20 base pairs, two closely related GPT models, and one-shot writing. In the larger task, GPT-5.5 uncued memories fail even on expected queries, GPT-5.4 struggles with the full exact cache, and several preregistered checks fail. It's hard to say what holds beyond the toy setting or what memory designers should take away besides testing shifted queries.
Questions for authors
-
Could the shifted-query evaluation be applied to an existing memory benchmark or system, such as LongMemEval or an agent memory pipeline? Even a small case would show whether the issue matters in practice.
-
In the larger task, GPT-5.5 uncued memories get only 52.5% even on expected queries. From the archived memories, is the needed information missing, wrong, or present but misread?
2 comments
Dear Reviewer,
We would like to once again sincerely thank you for your time and thoughtful feedback on our work.
As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.
Thank you again for your time and consideration.
Thank you for your careful review and for suggesting LongMemEval.
You are right that the single-feature cache in Table 2 already scores 50% after the shift. We do not claim that number is surprising. What the setup does not determine is whether a writer’s note will behave like that cache. In a separate experiment, we held the promise and 60 histories fixed: GPT-5.5 got 30/60 shifted decisions right with structured records and offers, versus 57/60 when both were written as prose.
The audit distinguishes errors that an end-to-end score counts as identical. All 302 lexical retrieval decisions with a nonzero retrieved sum agreed with its sign (repeated traces over 20 base pairs). In 44 of them, the retrieved events supported the opposite action from the full history. By contrast, GPT-5.4 made 29/80 errors in a separate control with a complete table of correct sums. The first finding points to event selection; in the second, the reader already had the right numbers.
Why did uncued GPT-5.5 score 52.5% on expected questions? In the archived run it estimated zero and declined 36 of 40 offers. That was correct in 18 cases; three of the other four were correct, giving 21/40. One note kept all 12 rewards and the decision rule but omitted which events had the queried feature. Another explicitly stored 0 where the correct sum was +1, and the reader reported 0. These are examples of missing and incorrect information, not a classification of all 19 errors.
On fresh 100-feature histories, the writer was told that any feature could matter. Raising only GPT-5.5’s writer effort improved expected accuracy from 23/40 to 39/40 and shifted accuracy from 22/40 to 39/40 (paired p < .001 for shifted). The reader and 500-character allowance were unchanged. GPT-5.4’s shifted change, 37/40 to 40/40, was not significant (p = .25). The gain on GPT-5.5’s expected questions cannot be explained by a broken promise: there was none.
Can the evaluation be applied to an existing benchmark? We constructed 50 histories from LongMemEval sessions, each with one question about what the user said and one about what the assistant said. GPT-5.5 wrote the notes before seeing either question and answered from them; GPT-5.4 judged the answers with the benchmark’s prompt.
The unfocused notes already favored user questions. The preregistered test was whether the user cue widened that gap: +16 percentage points (two-sided paired p = .056), short of .05. Across the four writer conditions, 176/200 notes exceeded the requested 200 words. In 16/50 histories, one question’s evidence session postdated the other question. This is a retrospective application of the test, not a confirmed replication in dialogue.