acceptodds
Withdrawn from ICLR 2027

Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term memory systems typically retrieve records similar to a new query. This assumes that a needed memory will resemble the query that needs it, but world knowledge can make dissimilar texts consequentially related: a tree-nut allergy should change an answer about macarons through their almond-flour ingredient. We call this failure mode the implicit-association blind spot and introduce InMind, a 125-task, expert-verified benchmark with paired controls that distinguish storage failure, missing bridging knowledge, and retrieval failure. With the decisive memory in context, GPT-5-mini gives the required reminder on 85.6% of indirect queries. Six vector, graph, and agentic memory systems reach at most 8.0% when success requires both recalling the fact and giving the reminder, despite reaching up to 100% direct-query accuracy. A larger embedding improves recall but leaves the gap intact. Conversely, a diagnostic probe that makes memory visible before the query arrives reaches 64.8%, locating the failure in query-conditioned retrieval and motivating routing: deciding which facts must remain visible.

Primary area:
datasets and benchmarks
Keywords:
agent memory, long-term memory, retrieval-augmented generation, memory evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FiNEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates hard tasks in LLMs. It also proposes a benchmark including 125 expert-verified tasks. Finally, a comprehensive evaluation demonstrates that larger embeddings do not close the gap.

Critical strengths

The proposed task is hard and significant for current LLMs. The collected data and benchmark are useful for future evaluation.

Critical weaknesses

The paper’s formulation deserves to be reorganized, as it contains too much unrefined content from AI tools, with overly abstract expressions that are hard to understand.

The proposed question likely already exists in current work, with its scope only expanded to hard tasks. This deserves clarification.

Questions for authors

There are some incorrect citations, such as MiniLM (Wang et al., 2020; Reimers & Gurevych, 2019), where the citation to Reimers & Gurevych (2019) does not correspond to MiniLM.

Some descriptions in the paper are directly generated by AI tools without manual adjustment, making them hard to understand. This is reflected in overly abstract expressions, while specific mechanisms are compressed by the narrative.

It is recommended that citations be placed close to the corresponding paper names rather than at the end of the sentence. For instance: “Indeed, HippoRAG 2, A-Mem, xMemory, and A-RAG all fail the example in Figure 1 (Gutierrez et al., 2025; Xu et al., 2025; Hu et al., 2026; Du et al., 2026).”

How does the proposed work compare with MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models? Do the proposed complex tasks fall within the scope of MM-OPERA?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FiNE

We appreciate the reviewer's suggestions on the paper's presentation. However, we disagree with the reviewer's other criticisms and suggestions. We look forward to a more careful, reasonable, and fair review from the reviewer.

Q1: The revised manuscript cites the all-MiniLM-L6-v2 model card

We thank the reviewer for pointing out the potential ambiguity in the citation. The full name of the model used in our paper is all-MiniLM-L6-v2, not the original MiniLM. It is a Sentence-Transformers sentence-embedding model further trained from MiniLM, and there is no separate paper introducing this specific checkpoint. We therefore originally cited both MiniLM and Sentence-BERT. We recognize that these citations could cause confusion. In the revised manuscript, we now cite the model's Hugging Face model card directly to identify the exact checkpoint and its source. We have also rechecked the remaining citations and found no similar ambiguities.

Q3: Citations now follow the corresponding system names

We also accept the suggestion to place citations immediately after the corresponding system names. We have checked and adjusted citation placement throughout the revised manuscript so that each system or work is directly associated with its citation, avoiding ambiguity from grouped citations at the end of a sentence.

We thank the reviewer for identifying these two issues and offering these suggestions. However, we disagree with the other points.

Q2: The claim of no manual revision is unsupported

We proactively disclosed our use of AI at the end of the paper, acknowledging AI-assisted writing while also explaining that AI was used only as an aid and that the manuscript underwent multiple rounds of human revision and review. The allegation of “without manual adjustment” is subjective and unsupported. If the reviewer did not understand specific passages, we would welcome concrete examples and would be happy to explain and adjust the wording.

Q4: MM-OPERA evaluates image associations; InMind evaluates memory systems

After internal discussion, we do not believe MM-OPERA falls within the scope of the related work that needs to be discussed here. MM-OPERA focuses on image understanding and reasoning, evaluating whether an MLLM can identify associations between two images. In contrast, our paper evaluates memory systems, which do not necessarily contain an LLM component, asking whether a memory system can present an LLM with the prior memories most useful for the current query. These are works in entirely different areas.

For these reasons, we also disagree with the weaknesses raised in the review.

Furthermore, the reviewer's Paper Summary contains a major error. Its first sentence is “The paper investigates hard tasks in LLMs.” This is incorrect. On the contrary, our tasks are very easy for LLMs (Section 4.3). Our paper evaluates memory systems, not LLMs. The reviewer's Paper Summary does not even mention memory.

The assessment of presentation is subjective. Reviewer Cmun rated our presentation “3: Clear and easy to follow.” Reviewer FiNE may find the presentation insufficient, and we have no objection to that difference of opinion. However, this cannot justify fundamentally misrepresenting the scope of our work. Our focus on memory systems is stated directly in both the abstract and introduction; this is not an issue of presentation. We can only conclude that the reviewer did not make an attempt to understand our work carefully, leading to such a major error.

We ask the reviewer to reassess our paper in a responsible and fair manner. If any points remain unclear to the reviewer, we would welcome specific examples and would be happy to clarify them. We look forward to a more careful, reasonable, and fair review.

Reviewer_CmunOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces InMind, a 125-task benchmark for testing whether memory systems apply a stored personal fact to a later query that requires background knowledge to connect the two. Each task includes a direct query and an indirect query. The evaluation measures whether the fact reaches the answerer’s context and whether the response gives the required reminder. Strict success requires both.

Across six memory systems and two embedding models, direct-query accuracy reaches 100%, but strict success on indirect queries reaches at most 8.0%. Providing the target memory directly yields 85.6%, while a simple always-visible memory state achieves 64.8%. The authors argue that these results expose a blind spot in query-conditioned retrieval.

Critical strengths
  1. The direct/indirect comparison captures a practical problem: a system can recall a user’s allergy when asked, yet fail to consider it when giving recipe advice. The benchmark makes this distinction clear and provides a useful test beyond explicit recall.

  2. The large retrieval gap appears across several memory systems and both embedding choices. Reporting recall separately from reminders makes the results easier to interpret. The human audit in Appendix F.2 also provides useful checks on the automatic judgments.

Critical weaknesses
  1. The paper draws broader conclusions about retrieval than the experiments support. The backbone control receives the target memory turn and query without the unrelated context supplied to the memory systems (Table 7). The always-in-state diagnostic also changes the memory updater and its prompt: it uses GPT-5-mini rather than the GPT-4o-mini used by most baselines, and explicitly prioritizes facts that affect advice or user safety (Appendix I). Although the answer model is shared, these comparisons do not isolate the benefit of correcting retrieval alone. A-RAG already uses an LLM to plan searches, so its failure with the default QA prompt also does not establish the broader limitation of query-conditioned search argued in Section 5.1.

  2. A positive strict-success score does not always demonstrate personalized use of memory. Appendix F.2 counts a generic allergen warning as a correct reminder. If the personal fact is also present in context, that warning can satisfy the strict score even if the model would give the same answer without the fact. The missing check is whether the answer changes appropriately when the personal fact is removed or replaced. This does not explain away the low retrieval recall, but it makes the claimed recovery from always-in-state memory harder to interpret.

Questions for authors
  1. What happens when the target fact is inserted into each system’s actual indirect-query context, with the other context, answer model, and answer prompt unchanged? How much of the performance gap does this recover?

  2. How does a retrieval policy that first identifies possible user constraints and then searches for them compare with the default A-RAG setup under the same search/token budget, without access to the target fact or gold bridge?

  3. On a subset of tasks, does the answer still receive a positive reminder score after the target fact is removed or replaced? Could the authors also include cases where the reminder is unnecessary to check for excessive warnings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Cmun

We thank the reviewer for the careful reading and valuable comments. We also appreciate the additional experiments the reviewer suggested: they substantially strengthen the reliability of our paper. We have completed these experiments and believe they address the weaknesses raised in the review and resolve the reviewer’s concerns about our conclusions.

Q1: Inserting the target fact raises score from 7% to 83%

We randomly selected 100 records with score R=W=S=0, inserted the target fact into their contexts, regenerated the answers, and evaluated them again. The results are reported in Table 20 of Appendix L.2. Reanswering before insertion gives W=7% and S=0%; after insertion, W=83% and S=83%.

Q2: Constraint-aware search helps but does not close the gap

Constraint-aware A-RAG reaches 29.6% strict success (S), up from 8.0% for the reported default A-RAG configuration. We append an instruction to A-RAG's default prompt to first reason about user constraints that could change the answer, then search the memory for supporting evidence (Appendix K.4). The success score remains significantly below full-history access (44.8%), the original always-in-state diagnostic (64.8%) and also the direct Query. This supports our conclusion that even with LLM-guided reasoning, the vast space of possible hypotheses makes it unlikely that the generated hypotheses cover the target fact, leaving the same underlying failure mode unresolved.

Q3: Removing the target fact lowers W score from 96% to 36%

We randomly selected 100 records with R=W=S=1, removed the target fact from their contexts, regenerated the answers, and evaluated them again (Appendix L.3, Table 20). Reanswering before removal gives W=96% and S=96%; after removal, W=36% and S=0%. Because the selected records originally had W=1, the sample may contain repeated instances of tasks prone to excessive warnings, making W higher than usual. Even so, W falls to just 36% after fact removal, showing that the target fact retains strong explanatory power for reminder generation.

Weaknesses 1 and 2: The experiments rule out the other possibilities the reviewer raised and support our conclusion

We believe the additional experiments in Q1 and Q2 address the reviewer’s concerns in Weakness 1. In Q1, the score approaches the backbone control even with unrelated context present. We also wish to clarify that always-in-state memory uses the same answering prompt as all other memory systems, as now explained in Appendix G.1.1. In Q2, even with a strongly leading prompt, A-RAG remains well below always-in-state memory. This shows that even when an LLM participates in retrieval, the vast space of possible hypotheses makes it unlikely that the generated hypotheses cover the target fact, leaving the same underlying failure mode unresolved.

We believe the experiment in Q3 also addresses the concern in Weakness 2. After removing the target fact, W drops from 100% (96% when reanswered) to 36%, showing that the target fact retains strong explanatory power for reminder generation.

We thank the reviewer again for suggesting these experiments and believe that the results have addressed the reviewer’s concerns.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We have responded to your comments and would like to ensure that we have addressed all of your concerns satisfactorily. Please let us know if you have any remaining questions or would like us to clarify anything further.

Best regards, The authors

Dear Reviewer,

We wanted to briefly follow up on our response, as the discussion period is nearing its end. We would like to ensure that we have addressed all of your concerns satisfactorily, and would be glad to clarify any remaining points. We understand you may have a busy schedule and appreciate the time you have spent reviewing our work.

Best regards, The authors

Reviewer_B9H3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agent memory systems usually retrieve facts similar to the current query, so a fact that becomes relevant only through world knowledge may never surface. The paper describes through an example: a "tree-nut allergy" matters for a "macaron recipe" because macarons use almond flour. The paper calls this the implicit-association blind spot.

Paper proposes a benchmark: "InMind" with 125 tasks (includes 113 source grounded and 12 tasks written by experts). Each task has a personal fact, a direct query, an indirect query, and a bridge explanation. These facts are injected into the LongMemEval trace and tested across various memory systems:

  • direct query accuracy D
  • recall R
  • required reminder W and strict success (inters. of R and W)

The paper evaluates the benchmark on Six Systems (A-RAG, xMemory, Mem0, A-Mem, HippoRAG 2, MemoryOS through GPT-5-mini as answerer and judge), each run with MiniLM or text-embedding-3-large, reach D = 76–100% but only R = 0.8–12% and S ≤ 8.0%.

The authors blame query-conditioned retrieval and name "routing" (which facts stay visible) as the open problem.

Critical strengths

S1. Clean design for locating failures, with audited judges. Each control isolates one cause:

  • the D (direct query) tests storage;
  • the target-only in-context control tests bridging knowledge;
  • R (recall) judged without the answer tests whether the fact surfaced;
  • the W (reminder) is judged without retrieved context;
  • Strict (S) requires both R (Recall) and W (Reminder).

Authors also present a 100-record human audit (Table 5) with the recall judge being reliable (97%, no false negatives), and that a single combined judge would over-credit generic warnings, which justifies the R/W split.

S2. Large, consistent effect across architectures under a uniform protocol. Table 1 covers all 12 system × embedding settings (vector, graph, agentic, OS-style):

D − R gaps are 70–97 points. S ≤ 8.0%. Even the best configuration's 95% upper bound (14.1%) is far below the backbone's lower bound (78.4%). Every injected fact goes through each system's own write and consolidation path (App. H).

Critical weaknesses
  1. The filter builds in the result, and the problem itself isn't new.

Tasks were kept only if BM25 and MiniLM-L6-v2 did not rate the memory as similar to the query, which dropped 700 of 1,000 candidates. The thresholds aren't given. The same scorers are then used as retrievers, so low recall is expected. Only text-embedding-3-large is an independent test (R = 2.4–12%). This shows the failure exists, not how common it is, so "false in general" (in Sec 2.1) seems overstated.

The paper also calls the assumption "never tested directly" (Sec 2.1), but prior work already shows memory/RAG systems missing relevant yet dissimilar user facts: for example: 'PrefEval' [1]: "I avoid … products containing silicones" → "What would you recommend for a daily makeup primer?" and LoCoMo-Plus [4], which uses a similar pipeline with psychological links. [1], [2] and [3] are not cited. The new parts from In-Mind are the controls: direct-query, source grounding, and the R/W/S split.

  1. Sec 5 conclusions rest on an unequal comparison and missing baselines.

(a) Unequal prompting for in-state config and existing systems. The always-in-state updater is told to store facts that "would change what advice you give (constraints, risks, needs)" (App. I), which is exactly InMind's fact type. Its answerer is also told to personalize. A-RAG uses its default QA prompt (Table 9); answer prompts for the other systems aren't reported, and none gets a comparable step such as asking an LLM which user facts might matter before retrieving. How much each approach recovers is untested under equal conditions.

(b) No full-history baseline. The model never sees the raw 47-session trace (~115k tokens, which fits in GPT-5-mini's context). So it's unclear whether the 64.8% comes from visibility or from curation. LoCoMo-Plus [4] and ConvoMem [3] show the full-context performance with (10–26%) and (63–82%) scores respectively on their own benchmarks, a full context performance baseline is missing.

(c) Routing is claimed but not tested. Sec 5.3 says "InMind directly evaluates this routing decision", but only one hand-built updater is evaluated. Its 200-line cap (Table 15) is probably never reached, so nothing is evicted. No existing routing system (e.g., MemGPT [5]) is evaluated.


References

[1] Zhao, S., Hong, M., Liu, Y., Hazarika, D., & Lin, K. (2025). Do LLMs recognize your preferences? Evaluating personalized preference following in LLMs. In International Conference on Learning Representations (ICLR), oral. arXiv:2502.09597.

[2] Li, X., Bantupalli, J., Dharmani, R., Zhang, Y., & Shang, J. (2025). Toward multi-session personalized conversation: A large-scale dataset and hierarchical tree framework for implicit reasoning. arXiv preprint arXiv:2503.07018.

[3] Pakhomov, E., Nijkamp, E., & Xiong, C. (2025). ConvoMem benchmark: Why your first 150 conversations don't need RAG. arXiv preprint arXiv:2511.10523.

[4] Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026). LoCoMo-Plus: Beyond-factual cognitive memory evaluation framework for LLM agents. arXiv preprint arXiv:2602.10715.

[5] Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560.

Questions for authors

Q1. How does a retriever that reasons first do on InMind? For example: one LLM call lists user facts that could matter for the query, then normal retrieval runs (text-embedding-3-large, same top-k). Or A-RAG prompted to check for user constraints first.

Q2.What is S for (i) the full 47-session trace in context, (ii) always-in-state with a neutral updater prompt ("record facts about the user"), and (iii) always-in-state with a cap tight enough to force eviction? How long did the memory file actually get?

Q3. Were the reported outputs generated on the final, expert-revised queries, also were the experts blind to system results? App. G says results were "re-scored … from the original full-run outputs", and App. A says expert revisions were mostly rewrites of the indirect query

Q4. What were the BM25/MiniLM thresholds, and how do the systems do on the 700 discarded candidates?

Q5. Who were the experts? How many were there, what was their expertise, how well did they agree, and how many tasks did they revise or reject? Can you also break the W-judge audit down by condition?

4 comments
AuthorsOpenReview ↗

Response to Reviewer B9H3 (1/2): Questions and Additional Experiments

We thank the reviewer for the detailed feedback. We added the requested controls and clarified the scope of our claims.

Q1: Constraint-aware A-RAG Result

We thank the reviewer for suggesting this control. On all 125 tasks, constraint-aware A-RAG reaches 29.6% strict success (S), up from 8.0% for the reported default A-RAG configuration. We append an instruction to A-RAG's default prompt to first reason about user constraints that could change the answer, then search the memory for supporting evidence (Appendix K.4). The success score remains significantly below full-history access (44.8%), the original always-in-state diagnostic (64.8%) and also the direct Query. This supports our conclusion that even with LLM-guided reasoning, the vast space of possible hypotheses makes it unlikely that the generated hypotheses cover the target fact, leaving the same underlying failure mode unresolved.

Q2: Full history baseline, neutral updater and force eviction

Full-history access reaches S=44.8%; the neutral always-in-state updater reaches S=68.8%, compared with 64.8% for the original focused updater (Table 18). The in-state result does not require an updater prompt that prioritizes safety or importance (Appendix K.3).

Under the 200-line cap, the neutral updater required 1,525 hard truncations across 4,883 updates (31.2%), and all 125 tasks encountered truncation. Final files were exactly 200 lines in 124/125 runs and averaged about 3,842 tokens (Appendix K.5, Table 19).

Q3: Tasks were finalized before evaluation

Yes. All reported answers used the final expert-revised queries, and the task reviewers did not see system outputs. The benchmark was fixed before formal evaluation. The re-scoring mentioned in Appendix G refers to a change in our evaluation procedure to reduce LLM-judge hallucinations: we replaced joint R/W scoring with separate judgments. This did not involve changing the tasks themselves. We apologize for the confusion and have clarified this in Appendix F.2.

Q4: Filtering thresholds

For each query, we initially required its target similarity to be at most twice its mean similarity to the fixed LME-s background under both BM25 and MiniLM cosine similarity, with the threshold computed separately for each measure. Slightly fewer than 300 candidates met both conditions, so we extended the selection in similarity-rank order to reach 300 (Appendix A).

Q5-1: Human annotator details

There were three reviewers, all master's students in computer science. The tasks primarily require common knowledge rather than specialist expertise and are supported by source references. The reviewers therefore worked independently on non-overlapping assignments of 100 tasks each, retaining 113 tasks in total.

Q5-2: The W judge breakdown

Table 6 in New Appendix F.2 breaks down the human audit by configuration.

AuthorsOpenReview ↗

Response to Reviewer B9H3 (2/2): Main Concerns and Clarifications

Weakness 1-1: Common memory systems from different paradigms fail on InMind

First, text-embedding-3-large was not used for filtering. Second, by “false in general,” we meant “false across different memory paradigms.” Embeddings are only a small component of these memory systems. Their more central components include graph structures, episodic/semantic memory extraction, and agentic reasoning. We use two embedding configurations precisely to show that embeddings are not the main reason these memory paradigms fail.

Weakness 1-2: Prior work observed related failures, but we are the first to isolate and quantify them

Prior work has observed related failures, but this is the first study to isolate and quantify this failure with paired controls. In previous work, implicit-association failures are typically entangled with other failure modes. A failed task cannot rule out insufficient LLM capability or failure to store the information. Because those tasks cannot exclude these confounds, implicit association remains only a hypothesized explanation, not one that can be conclusively established. In addition, we have added the related work the reviewer identified to Related Work.

Weakness 2-a: The prompting is fair

Actually, all systems receive the same external answering instruction, now listed separately in Appendix G.1.1. We thank the reviewer for pointing out where our paper was unclear. A-RAG uses its own default prompt templates internally, but it receives the same external instruction as the other systems.

Weakness 2-b: We now have a full-history result

We added a result that passes the entire 47-session trace, including the target fact, to the answerer. Appendix K.2 describes the setup; Q2 reports the result. Full-history access achieved 44.8% strict success (S), well above the scores of the evaluated retrieval-based memory systems. This further supports our conclusion that target-fact visibility is an important factor behind the performance gap.

Weakness 2-c: We already tested a routing system: MemoryOS

First, MemoryOS, which we tested in this paper, is a routing system. It did not perform particularly well, suggesting that some engineering details need adjustment. As a benchmark, InMind is designed precisely to evaluate such adjustments. Tuning the system itself is not central to our work.

We appreciate the reviewer’s feedback and believe that the additional controls and clarifications have addressed the concerns raised.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We have responded to your comments and would like to ensure that we have addressed all of your concerns satisfactorily. Please let us know if you have any remaining questions or would like us to clarify anything further.

Best regards, The authors

Dear Reviewer,

We wanted to briefly follow up on our response, as the discussion period is nearing its end. We would like to ensure that we have addressed all of your concerns satisfactorily, and would be glad to clarify any remaining points. We understand you may have a busy schedule and appreciate the time you have spent reviewing our work.

Best regards, The authors

Comments on the paper

1 comment
Area_Chair_8Up5OpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.