- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a benchmark, , to test whether memory systems can connect a conversational project cue to answer-bearing evidence stored in a larger document collection. The setup uses persona-specific scholarly libraries, synthetic anchor conversations, and four QA types: Technical Detail, Section Retrieval, Document Browsing, and Multihop Reasoning. The conversation is intended to tell the system what the user is referring to, while the document corpus contains the actual answer. The evaluation covers16 memory/retrieval systems plus a long-context backbone. Its main finding is that, under corpus-first ordering, 15 of 16 memory methods underperform the shared OAI-4o-mini backbone on L1 accuracy.
Critical strengths
None
Critical weaknesses
W1 Fatal Weakness: Fundamental mismatch between the claimed personalization problem and the task actually evaluated (Primary Blocker)
My fundamental concern is with the central problem formulation and positioning of the paper. I do not find the benchmark to be evaluating personalization in the sense that user-specific information changes what answers or behavior are appropriate. Instead, the persona mainly scopes the relevant project/library context and helps resolve which documents should be searched. Once this reference is resolved, the target answer is an ordinary document-grounded fact.
Central Evidence
- Construction doc-QA questions from document corpus
- An anchor containing a codename or topical cue is then associated with that instance
- Final query is produced by combining reference with pre-existing document question
Therefore, the conversation resolves project reference, whereas the document corpus supplies answer-bearing facts. User/persona primarily determines the relevant evidence scope rather than subjective appropriate answers. For example, suppose Alice refers to a relevant topic through a project named GreenShapes and Bob refers to the same scientific topic through another project name. If both contexts ultimately point to the same fact that 3D printing enables relevant dynamic forms, the correct answer remains '3D PRINTING'. The personalized component changes the route used to locate the evidence, not the scientific answer itself. I would regard a genuinely personalized task as one where the same or closely matched query can require a different appropriate response because users differ in preferences, goals, expertise, constraints, history, or state. The present benchmark does not appear to test this distinction. only establishes that different personas are associated with different collections; it does not establish that the correct target response is personalized.
Prior Works and SOTA deployed LLMs already cover project-scoped reference resolution
The motivating scenerio is very close to the workspace/project abstraction used in systems such as ChatGPT, Claude projects: a project contains its own files, conversations and cotexts, and a question referring to that project is answered using corresponding project materials. For example, if I ask a question about project A, the system should naturally use Project A's files rather than files belonging to an unrelated project B. I view this as project/reference scoping. Unless experiments reveal something very interesting and fundamental regarding such deployed settings (like whether the new benchmark exposes why the current user-specific projects and the followup discussions are not trustworthy), I do not find there is any specific contribution of this benchmark.
Empirical results do not resolve the issue
The experimental findings revolve around:
- memory systems performing worse than shared backbone
- positioning of anchor matters
- comnpression of evidence and its effect
and so on.
These findings do not demonstrate that is the source of difficulty. Therefore,
$ \text{interesting cross-source memory/RAG failure} \neq \text{evidence of a personalization failure}. $
Why this issue is Decision Critical
The issue is not a mere terminology or re-scoping issue that can be addressed via title or wording. Under the current construction, I do not see evidence that the benchmark evaluates personalization beyond persona/project-scoped reference resolution and evidence retrieval. Resolving this concern would therefore require substantial reframing around cross-source or project-scoped evidence integration, or additional evaluation where user-specific information genuinely changes the appropriate target response and where the claimed gap is demonstrated against systems that instantiate the project/workspace setting used to motivate the paper.
Questions for authors
I have no additional questions
2 comments
Sorry for not being able to revise my score
Thank you for the detailed response and for clarifying the revised positioning. I appreciate that the authors have acknowledged the distinction raised in my review. However, I am still not convinced that the response resolves my main concern, for two reasons.
1. The proposed reframing is substantial, rather than a clarification of the original problem.
My original concern was that the benchmark evaluates project/persona-scoped evidence resolution rather than personalization. The response largely agrees with this observation and proposes to reposition the work around project-conditioned document-evidence resolution.
I see this as a substantial change to the central problem statement, not simply a clarification of its terminology. Such a change would require reconsidering the motivation, related literature, contribution claims, and interpretation of the results under a different problem formulation. While I appreciate the authors' willingness to revise, I do not think that accepting a different problem positioning during rebuttal is, by itself, sufficient justification for moving my score from clear reject to weak reject. In my view, the revised framing would require a more complete reassessment of the work.
2. Under the revised framing, the main empirical findings appear to overlap considerably with established literature.
My second concern is what new scientific understanding the benchmark provides once the personalization claim is removed. Several of the observations emphasized in the response have already been studied in the long-context, RAG, compression, and LLM-memory literature.
| Finding emphasized in the submission/rebuttal | Relevant prior work | Finding already established |
|---|---|---|
| Performance changes when the conversational anchor and document corpus are presented in different orders. | Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2024) | LLM performance can depend strongly on where relevant information appears within a long context. While the source-order experiment is different, the broader phenomenon of positional sensitivity is already established. |
| Several memory/retrieval systems perform worse than directly providing the available context to the backbone. | Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (Li et al., 2024) | When sufficient context capacity is available, direct long-context inference can outperform retrieval-augmented approaches, despite their computational advantages. |
| Compression and summarization can remove answer-bearing evidence and reduce answer accuracy. | Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs (Cao et al., 2024) | Loss of key information during compression can substantially degrade QA performance, motivating methods that explicitly preserve query-relevant evidence. |
| Preserving original evidence can be more reliable than repeatedly rewriting or summarizing memory. | AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models (Shen et al., 2026) | Repeated memory rewriting and summarization can dilute contextual details. The work separates retrieval anchors from preserved original context to support more reliable memory retrieval and generation. |
I am not suggesting that these papers evaluate exactly the same benchmark setting. However, the underlying findings concerning positional sensitivity, retrieval versus full-context trade-offs, information loss through compression, and the importance of retaining original evidence are not new.
The additional multi-backbone experiments provide further evidence that these behaviors occur within the proposed benchmark. However, I do not see how they establish a substantially new empirical insight under the revised project-conditioned evidence-resolution setting, beyond demonstrating already recognized failure patterns in another evaluation environment.
Overall, the response acknowledges my original concern but addresses it primarily through a different problem framing. Under that framing, I remain unconvinced about the novelty of the central empirical findings. Therefore, my assessment remains unchanged, and I retain my original score.
Official Response to Reviewer sLVS
Response to Reviewer sLVS
Thank you for pinpointing this core conceptual distinction. Below we adopt your suggested reframing across the manuscript, compare flat and structured memory baselines within bounded persona workspaces across three backbones, and state our open-architecture scope.
Reply to W1
1. Task formulation: project-conditioned document-evidence resolution. Your
GreenShapesexample captures the exact structure of our benchmark: chat history identifies which project, study, or comparison is referenced, while the scientific ground-truth answer (3D PRINTING) remains invariant once that referent is resolved. Following your recommendation for substantial reframing, we revise the Abstract, Section 1 (Table 1), Section 2.1, Section 4, Section 6, Appendix A.1 (Table 5), and Appendix E to define the evaluated capability explicitly as project-conditioned document-evidence resolution and integration, removing claims that user state alters ground-truth answers to identical resolved queries and noting counterfactual preference-driven QA as future work.2. Multi-backbone comparison within bounded persona workspaces. Each released persona context already forms a bounded research workspace of 4–28 paper blocks (196 blocks across 10 personas, 72k–168k tokens with chats in Appendix B.2). Across all n=266 questions on three backbones (Section 5.2, Appendix C.10, Figure 7 and Tables 22–24), structured entity-graph indexing surpasses flat workspace baselines on L2 across all three backbones and on L1 under newer backbones:
OAI-4o-mini reports three-run means (Table 3), while DeepSeek-V4.1-Flash and Hy4 report complete single-run matrices (Appendix C.10, Tables 22–23).
Even inside a 4–28 paper workspace, flat reading (
Full context), lexical retrieval (BM25), and recursive summarization (RAPTOR[2]) often miss study-specific qualifiers—across 3,974 method–query instances, L1 drops from 63.3% to 39.3% when normalized gold-answer strings are absent from generator inputs (Section 5.4)—whereas passage-preserving entity-graph stores (Cognee,HippoRAG-v2[1]) achieve statistically significant gains overFull contextonDeepSeek-V4.1-FlashandHy4(Holm pH=0.032 and 0.037 on L1, pH<0.001 on L2).3. Open-architecture scope. PerDocBench evaluates 16 open, reproducible memory architectures under controlled corpus and prompt protocols (Appendices C.10–C.13, Appendix E), complementing proprietary commercial workspace interfaces whose closed routing layers cannot be ablated.
References
[1] Gutiérrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (HippoRAG 2). arXiv:2502.14802, 2025.
[2] Sarthi et al. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. ICLR, 2024.
[3] DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017, 2026.
[4] Tencent. Tencent Launches Hy4, Joining the Ranks of the World's Leading Open-Source AI Models. Official release & public API: https://openrouter.ai/tencent/hy4, 2026.