- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates hard tasks in LLMs. It also proposes a benchmark including 125 expert-verified tasks. Finally, a comprehensive evaluation demonstrates that larger embeddings do not close the gap.
Critical strengths
The proposed task is hard and significant for current LLMs. The collected data and benchmark are useful for future evaluation.
Critical weaknesses
The paper’s formulation deserves to be reorganized, as it contains too much unrefined content from AI tools, with overly abstract expressions that are hard to understand.
The proposed question likely already exists in current work, with its scope only expanded to hard tasks. This deserves clarification.
Questions for authors
There are some incorrect citations, such as MiniLM (Wang et al., 2020; Reimers & Gurevych, 2019), where the citation to Reimers & Gurevych (2019) does not correspond to MiniLM.
Some descriptions in the paper are directly generated by AI tools without manual adjustment, making them hard to understand. This is reflected in overly abstract expressions, while specific mechanisms are compressed by the narrative.
It is recommended that citations be placed close to the corresponding paper names rather than at the end of the sentence. For instance: “Indeed, HippoRAG 2, A-Mem, xMemory, and A-RAG all fail the example in Figure 1 (Gutierrez et al., 2025; Xu et al., 2025; Hu et al., 2026; Du et al., 2026).”
How does the proposed work compare with MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models? Do the proposed complex tasks fall within the scope of MM-OPERA?
Response to Reviewer FiNE
We appreciate the reviewer's suggestions on the paper's presentation. However, we disagree with the reviewer's other criticisms and suggestions. We look forward to a more careful, reasonable, and fair review from the reviewer.
Q1: The revised manuscript cites the all-MiniLM-L6-v2 model card
We thank the reviewer for pointing out the potential ambiguity in the citation. The full name of the model used in our paper is all-MiniLM-L6-v2, not the original MiniLM. It is a Sentence-Transformers sentence-embedding model further trained from MiniLM, and there is no separate paper introducing this specific checkpoint. We therefore originally cited both MiniLM and Sentence-BERT. We recognize that these citations could cause confusion. In the revised manuscript, we now cite the model's Hugging Face model card directly to identify the exact checkpoint and its source. We have also rechecked the remaining citations and found no similar ambiguities.
Q3: Citations now follow the corresponding system names
We also accept the suggestion to place citations immediately after the corresponding system names. We have checked and adjusted citation placement throughout the revised manuscript so that each system or work is directly associated with its citation, avoiding ambiguity from grouped citations at the end of a sentence.
We thank the reviewer for identifying these two issues and offering these suggestions. However, we disagree with the other points.
Q2: The claim of no manual revision is unsupported
We proactively disclosed our use of AI at the end of the paper, acknowledging AI-assisted writing while also explaining that AI was used only as an aid and that the manuscript underwent multiple rounds of human revision and review. The allegation of “without manual adjustment” is subjective and unsupported. If the reviewer did not understand specific passages, we would welcome concrete examples and would be happy to explain and adjust the wording.
Q4: MM-OPERA evaluates image associations; InMind evaluates memory systems
After internal discussion, we do not believe MM-OPERA falls within the scope of the related work that needs to be discussed here. MM-OPERA focuses on image understanding and reasoning, evaluating whether an MLLM can identify associations between two images. In contrast, our paper evaluates memory systems, which do not necessarily contain an LLM component, asking whether a memory system can present an LLM with the prior memories most useful for the current query. These are works in entirely different areas.
For these reasons, we also disagree with the weaknesses raised in the review.
Furthermore, the reviewer's Paper Summary contains a major error. Its first sentence is “The paper investigates hard tasks in LLMs.” This is incorrect. On the contrary, our tasks are very easy for LLMs (Section 4.3). Our paper evaluates memory systems, not LLMs. The reviewer's Paper Summary does not even mention memory.
The assessment of presentation is subjective. Reviewer Cmun rated our presentation “3: Clear and easy to follow.” Reviewer FiNE may find the presentation insufficient, and we have no objection to that difference of opinion. However, this cannot justify fundamentally misrepresenting the scope of our work. Our focus on memory systems is stated directly in both the abstract and introduction; this is not an issue of presentation. We can only conclude that the reviewer did not make an attempt to understand our work carefully, leading to such a major error.
We ask the reviewer to reassess our paper in a responsible and fair manner. If any points remain unclear to the reviewer, we would welcome specific examples and would be happy to clarify them. We look forward to a more careful, reasonable, and fair review.