acceptodds
Under review as a conference paper at ICLR 2027

Beyond In-Task Accuracy: Evaluating Task-Specialized Memory under Relevance Shifts

Abstract

High accuracy on anticipated queries does not establish whether an LLM memory remains useful when different aspects of past experience become relevant. We isolate this evaluation gap by changing a memory writer's announced task while holding the history and decision rule fixed. In a controlled feature–reward task with exact scoring and reward-reversed histories, the writer is either promised that one feature dimension will matter or told that any dimension may matter. We freeze its memory and evaluate both announced and shifted queries. In the small task, cued memories from GPT-5.5 and GPT-5.4 achieve 100% in-task accuracy but 50.0–51.7% after a relevance shift. Uncued memories achieve 100% on both without exhausting the same 100-word allowance. Thus, identical in-task scores conceal different cross-query capabilities even when a broader useful record fits. The shift deliberately violates the cued writer's promise: specialization is appropriate for the announced task, not evidence of irrational forgetting. A separate 100-dimension task exposes model-dependent encoding and reading limitations rather than a uniform shift effect. A post-hoc retrieval audit distinguishes another source of error: all 302 non-tie lexical actions follow the selected reward sum, including 44 wrong full-history answers. Together, these results support evaluating anticipated-task accuracy, relevance-shift performance, and evidence use separately, rather than treating success on the anticipated task as a sufficient measure of memory quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, agent memory, memory evaluation, relevance shift, evidence retrieval, decision-making

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_9uSkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that memories shouldn't be judged only on the questions they were written for. In a controlled reward-summing task, cued memories get 100% on expected queries but drop to about 50% when the query shifts, while uncued memories stay at 100%. A larger task and a retrieval audit show that other errors come from how models encode and read memory, or from incomplete retrieved evidence.

Critical strengths
  1. The main comparison is cleanly controlled. History, decision rule, and budget are all fixed, and only the writing instruction changes. Two memories with identical in-task scores end up very different on shifted queries, and the uncued memories don't use more space. This makes a clear case that in-task accuracy alone can rank quite different memories as equal.

  2. The two-reference scoring is a useful idea. By checking each answer against both the full history and the events the reader actually saw, the paper separates wrong evidence from wrong reading. This could be reused when evaluating memory or RAG systems more broadly.

Critical weaknesses
  1. Most findings follow directly from the setup. The cued writer is told every later query will be about one feature, so it reasonably stores only that feature. The programmatic cued cache in Table 2 gets exactly the same 50% on shifted queries, so the LLM result adds little beyond what the setup already implies. The retrieval audit is similar. A reader given a subset of events follows the sum of those events, which is again expected.

  2. The evidence is narrow, and beyond the small task the results are mixed. The tasks use 6 or 12 synthetic events, with 30 and 20 base pairs, two closely related GPT models, and one-shot writing. In the larger task, GPT-5.5 uncued memories fail even on expected queries, GPT-5.4 struggles with the full exact cache, and several preregistered checks fail. It's hard to say what holds beyond the toy setting or what memory designers should take away besides testing shifted queries.

Questions for authors
  1. Could the shifted-query evaluation be applied to an existing memory benchmark or system, such as LongMemEval or an agent memory pipeline? Even a small case would show whether the issue matters in practice.

  2. In the larger task, GPT-5.5 uncued memories get only 52.5% even on expected queries. From the archived memories, is the needed information missing, wrong, or present but misread?

2 comments

Thank you for your careful review and for suggesting LongMemEval.

You are right that the single-feature cache in Table 2 already scores 50% after the shift. We do not claim that number is surprising. What the setup does not determine is whether a writer’s note will behave like that cache. In a separate experiment, we held the promise and 60 histories fixed: GPT-5.5 got 30/60 shifted decisions right with structured records and offers, versus 57/60 when both were written as prose.

The audit distinguishes errors that an end-to-end score counts as identical. All 302 lexical retrieval decisions with a nonzero retrieved sum agreed with its sign (repeated traces over 20 base pairs). In 44 of them, the retrieved events supported the opposite action from the full history. By contrast, GPT-5.4 made 29/80 errors in a separate control with a complete table of correct sums. The first finding points to event selection; in the second, the reader already had the right numbers.

Why did uncued GPT-5.5 score 52.5% on expected questions? In the archived run it estimated zero and declined 36 of 40 offers. That was correct in 18 cases; three of the other four were correct, giving 21/40. One note kept all 12 rewards and the decision rule but omitted which events had the queried feature. Another explicitly stored 0 where the correct sum was +1, and the reader reported 0. These are examples of missing and incorrect information, not a classification of all 19 errors.

On fresh 100-feature histories, the writer was told that any feature could matter. Raising only GPT-5.5’s writer effort improved expected accuracy from 23/40 to 39/40 and shifted accuracy from 22/40 to 39/40 (paired p < .001 for shifted). The reader and 500-character allowance were unchanged. GPT-5.4’s shifted change, 37/40 to 40/40, was not significant (p = .25). The gain on GPT-5.5’s expected questions cannot be explained by a broken promise: there was none.

Can the evaluation be applied to an existing benchmark? We constructed 50 histories from LongMemEval sessions, each with one question about what the user said and one about what the assistant said. GPT-5.5 wrote the notes before seeing either question and answered from them; GPT-5.4 judged the answers with the benchmark’s prompt.

Reader inputUser questionsAssistant questions
User-focused note46/507/50
Assistant-focused note37/5022/50
Unfocused note41/5010/50
Mem0 fact-extraction prompt only49/509/50
Full conversation50/5050/50

The unfocused notes already favored user questions. The preregistered test was whether the user cue widened that gap: +16 percentage points (two-sided paired p = .056), short of .05. Across the four writer conditions, 176/200 notes exceeded the requested 200 words. In 16/50 histories, one question’s evidence session postdated the other question. This is a retrospective application of the test, not a confirmed replication in dialogue.

Dear Reviewer,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

Reviewer_hvkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study varies the task announced to a memory writer, freezes its note, and tests expected and shifted queries over the same history. Identical in-task accuracy can hide different query coverage. A separate retrieval audit checks answers against both full-history and selected evidence.

Critical strengths
  1. Exact scoring and paired reward reversals make the central result easy to interpret. Successful uncued notes within the same allowance rule out a necessary storage shortage in the small task.
  2. The authors distinguish evidence omission from failure to follow evidence, and report the larger-task replication failures. The conclusions are appropriately cautious about these differences.
Critical weaknesses
  1. The shifted test contradicts an explicit promise that only one feature matters. A specialized note failing this test is unsurprising, so the contribution beyond illustrating task mismatch is limited. A control with uncertain future relevance would make the evaluation lesson more consequential.
  2. The experiments cover short synthetic histories and two closely related models. The larger task changes event count, dimensions, and budget units together, with substantially different results between models. It therefore provides limited evidence about scaling or persistent agent memory.
Questions for authors
  1. Suggested control: does the gap persist when writers are told that other features may occasionally matter?
  2. Clarification: which conclusion from the larger task holds across both models independently of their chosen encoding?
2 comments

Thank you for your review and comments and suggesting a direct test of what happens when the writer is told that other features might matter.

The result is about the test, not a failure to hedge. A writer promised that only one feature will matter has a good reason to specialize. The question is whether a perfect score on that feature tells an evaluator what else the note can support. In the new small-task control, the strict-promise and hedged writers each scored 120/120 on expected questions. When we asked about another feature of those same histories, their frozen notes scored 60/120 and 120/120, respectively.

Does the gap persist under uncertain relevance? No. The comparison above used 60 reward-reversed pairs. We changed only the writer’s instruction, from a guarantee to a warning that most offers would concern one feature but others might occasionally matter. The counts held separately for GPT-5.5 and GPT-5.4; the reader, decision rule, and 100-word allowance stayed fixed. With no preferred feature, each model also scored 120/120 on both questions. In a separate control with that same open instruction, ten earlier questions favored color over category 8:2, or the reverse. Both models scored 80/80 on each feature under either history. Both features fit in the note, so this does not test hedging when the writer must omit something.

What holds across models as the task grows? We varied the number of features (4, 8, 16, 32, 64, 100), fixing history length at 12 events and the allowance at 500 characters, with 20 reward-reversed pairs per size. At every size, each model’s strict-promise notes scored 40/40 expected and 20/40 shifted. Given the raw events or all matching events, each scored 40/40 on both. These comparisons require no assumption about how to decode a model’s note.

The two models did not show a common capacity limit. At 100 features, raising only the uncued writer’s reasoning effort moved shifted accuracy from 22/40 to 39/40 for GPT-5.5 (paired p < .001), versus 37/40 to 40/40 for GPT-5.4 (p = .25). GPT-5.4 also fell below the 90% control threshold on at least one query type when reading a complete table of correct sums at every tested size from eight features onward.

Dear Reviewer,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

Reviewer_UpTVOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Please ignore my placeholder review below. I do not have sufficient expertise to assess this submission reliably and have alerted the AC.

Critical strengths

n/a

Critical weaknesses

n/a

Questions for authors

n/a

Reviewer_5DkoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates how an existing LLM memory survives a relevance shift, that is, a change in the query it is later asked to answer. The authors define three objects: a history H (sequence of past events), a memory M (compressed representation), and a query q that can be either expected (in-task) or shifted. The central question is: if a memory answers the expected query perfectly, how well does it answer a different query about the same history? To study this, the authors design a controlled binary decision task with synthetic histories and compare memories written with explicit instruction about the relevant dimension (cued) versus memories written without that instruction (uncued). Experiments are conducted with GPT-5.5 and GPT-5.4 across two task configurations. Results show that cued memories achieve 100% accuracy on the expected query but drop to ~50% (chance level) on a shifted query, while uncued memories maintain 100% on both in the small-task setting. A retrieval experiment further shows that field-pair BM25 outperforms word BM25 and dense retrieval, and a selected-evidence audit demonstrates that observed errors stem from retrieving incorrect evidence rather than from misreading the available evidence.

Critical strengths

S1: The paper is well-written and easy to follow; figures and results are clearly presented.

S2: It addresses a genuinely interesting and timely problem for the memory evaluation community.

S3: The related work section provides an up-to-date and well-organized review of the area.

S4: The authors are transparent about the limitations of their study.

Critical weaknesses

W1. Limited experimental scope and questionable generalizability. Experiments are conducted on highly controlled synthetic histories (at most 12 events, binary dimensions, rewards drawn from a small discrete set) using only two closely related proprietary models. While the authors acknowledge this in the Limitations section, they do not discuss whether the observed phenomenon might change qualitatively in real-world settings, where events are free-form natural language and shifted queries may differ structurally from expected ones.

W2. Undefined terminology in figures. Figure 1 uses the terms Cued, Uncued, and Oracle in its legend before they are formally defined in the text, and without sufficient description within the figure itself. Figures should be self-contained, or at minimum include captions that briefly explain each condition, particularly in a paper whose central contribution rests on this very distinction.

W3. Limited reproducibility. No code or scripts are shared to allow independent replication of the experiments. Given that the paper proposes an evaluation methodology intended for adoption by the community, the absence of reproducible materials substantially limits its practical utility.

Questions for authors

I have no specific questions for the authors. My doubts and concerns regarding the paper are described in the Weaknesses section above.

2 comments

Thank you for your careful review and asking whether the result holds when the same events are described in prose.

Your scope concern draws a useful distinction: the paper proposes a way to test what a note can answer, not a prediction that changing the question always causes a 50-point drop. We tested two changes you identified. Rewording the same histories substantially reduced the drop; asking a question that needs a combination of features exposed a limitation even in a table containing every single-feature total.

Presentation and query structure. On the same 60 six-event histories, we compared structured records and offers with lightly paraphrased first-person events and natural-language offers. Under the same single-feature promise, shifted accuracy went from 30/60 to 57/60 for GPT-5.5 and from 31/60 to 46/60 for GPT-5.4. One color-organized note still recorded the category and reward of each green item. Prioritizing color did not erase category. Both the events and the offers were reworded, so we cannot attribute the difference to either change alone.

A question about “red and food” needs information that separate totals for red items and food items do not generally preserve. On the structured histories, the table of all single-feature totals supported 60/60 correct decisions on each single-feature query type, but only 44/60 joint decisions with GPT-5.5 and 40/60 with GPT-5.4. When writers knew combinations were possible but not which would be asked, their notes supported 60/60 correct joint decisions for each model in both renderings. The paraphrases retained a controlled vocabulary, and the two models are from one family. In our 50-history LongMemEval case study, the preregistered cue comparison did not reach significance (p = .056). We do not claim generalization to unrestricted dialogue or other model families.

Figure 1. “Cued” means the writer was promised category-only offers; “uncued” means any feature could matter. The oracle accepts if the matching rewards in the full history sum to a positive value. The six events are yellow/object +1, yellow/animal +1, blue/animal −2, red/food +2, red/animal −1, and blue/object −1. The animal total is −2; both notes lead the reader to decline. The red total is +1; the category-only note leads to an estimate of zero and a decline, while the uncued note leads to +1 and acceptance. Negating every reward reverses both correct actions without changing either question.

For readers who want to repeat the audit, Appendix A describes how we generated the histories and gives the seeds and small-task writing instructions; Appendix D.1 gives the scoring rule, including how ties are handled.

Dear Reviewer,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

Comments on the paper

1 comment

We thank the reviewers for pressing us on three questions: what the evaluation shows beyond instruction following, where incorrect decisions arise, and whether the findings extend beyond the original task.

1. What perfect in-task accuracy leaves untested

A writer told that only one feature will matter has good reason to specialize. Our question is what an evaluator can infer from a perfect score on that feature. Following a reviewer’s suggestion, we changed the instruction from a guarantee that every later offer would concern one feature to a warning that most would, but another feature might matter.

For GPT-5.4 and GPT-5.5 separately, both conditions scored 120/120 on expected questions. When we froze the notes and asked about another feature of the same histories, the scores were 60/120 under the guarantee and 120/120 under the warning. The 60/120 has a precise interpretation: in all 60 reward-reversed pairs, the correct action flipped, but the reader using the guarantee-condition notes gave the same answer. Both features fit within 100 words, and the writers hedged when warned. It was the in-task score that failed to distinguish the two sets of notes.

2. Distinguishing errors in writing, retrieval, and reading

We varied the number of features from 4 to 100 while keeping history length at 12 events and the memory allowance at 500 characters. Under the strict guarantee, both models scored 40/40 on expected questions and 20/40 on shifted questions at every size. Without a preferred feature, increasing only the writer’s reasoning effort at 100 features raised GPT-5.5’s shifted accuracy from 22/40 to 39/40 (paired p < .001). GPT-5.4’s change from 37/40 to 40/40 was not significant (p = .25).

The audit also separated two errors that look identical in an accuracy score. In 44 retrieval decisions, the reader correctly followed the retrieved events, but those events supported the opposite action from the full history. In a separate control, GPT-5.4 made 29/80 errors even with a complete table of correct sums. One result points to which evidence reached the reader; the other points to how the reader used it.

3. What changes with presentation and question type

The strict guarantee did not make a 50% score inevitable. In a separate comparison on the same 60 histories, GPT-5.5’s shifted accuracy was 30/60 with structured events and offers versus 57/60 with both written as prose (GPT-5.4: 31/60 versus 46/60). One prose note illustrates the difference: it grouped items by color but still recorded the category and reward of each green item. It respected the cue without discarding all the evidence needed for a category question.

Question structure mattered too: separate totals for “red” and “food” do not generally determine the total for items that are both. In our retrospective LongMemEval case study, the preregistered comparison did not establish that a user-focused cue widened the gap between user and assistant questions (p = .056). We therefore do not claim a universal penalty for specialization.

Our recommendation is to report what else the same frozen memory can answer alongside its in-task accuracy, and to check wrong decisions against the evidence the reader actually received. The individual replies give the detailed comparisons for each reviewer’s questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.