acceptodds
Under review as a conference paper at ICLR 2027

PerDocBench: Evaluating Cross-Source Evidence Integration in Personalized Memory Systems

Abstract

Personalized memory agents are increasingly expected to integrate a user’s conversations with private documents, yet most evaluations still rely on homogeneous chat streams that can reward benchmark-specific retrieval shortcuts. This makes progress difficult to interpret: strong performance may reflect adaptation to benchmark artifacts rather than robust use of heterogeneous personal evidence. We introduce PerDocBench, a cross-source benchmark for evaluating whether memory systems can connect conversational cues with document-grounded facts in persona-specific scholarly contexts. Unlike chat-only benchmarks, PerDocBench targets questions that require evidence to be identified, retained, and integrated across structurally different sources. Experiments with 16 representative memory methods under a shared backbone reveal a negative-utility pattern: averaged over three runs, 15 of 16 methods recover the requested fact less often than the backbone. These results suggest that current memory systems often degrade cross-source evidence integration in scholarly document-centered personalization. An anchor-reordering ablation yields a median L1 gain of 7.1% across systems and motivates anchor-first, evidence-preserving memory construction as a future design direction.

Primary area:
datasets and benchmarks
Keywords:
Personalized memory agents, Memory benchmarks, Cross-source evidence integration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sLVSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a benchmark, PerDocBench\texttt{PerDocBench}, to test whether memory systems can connect a conversational project cue to answer-bearing evidence stored in a larger document collection. The setup uses persona-specific scholarly libraries, synthetic anchor conversations, and four QA types: Technical Detail, Section Retrieval, Document Browsing, and Multihop Reasoning. The conversation is intended to tell the system what the user is referring to, while the document corpus contains the actual answer. The evaluation covers16 memory/retrieval systems plus a long-context backbone. Its main finding is that, under corpus-first ordering, 15 of 16 memory methods underperform the shared OAI-4o-mini backbone on L1 accuracy.

Critical strengths

None

Critical weaknesses

W1 Fatal Weakness: Fundamental mismatch between the claimed personalization problem and the task actually evaluated (Primary Blocker)

My fundamental concern is with the central problem formulation and positioning of the paper. I do not find the benchmark to be evaluating personalization in the sense that user-specific information changes what answers or behavior are appropriate. Instead, the persona mainly scopes the relevant project/library context and helps resolve which documents should be searched. Once this reference is resolved, the target answer is an ordinary document-grounded fact.

Central Evidence

  • Construction doc-QA questions from document corpus
  • An anchor containing a codename or topical cue is then associated with that instance
  • Final query is produced by combining reference with pre-existing document question

Therefore, the conversation resolves project reference, whereas the document corpus supplies answer-bearing facts. User/persona primarily determines the relevant evidence scope rather than subjective appropriate answers. For example, suppose Alice refers to a relevant topic through a project named GreenShapes and Bob refers to the same scientific topic through another project name. If both contexts ultimately point to the same fact that 3D printing enables relevant dynamic forms, the correct answer remains '3D PRINTING'. The personalized component changes the route used to locate the evidence, not the scientific answer itself. I would regard a genuinely personalized task as one where the same or closely matched query can require a different appropriate response because users differ in preferences, goals, expertise, constraints, history, or state. The present benchmark does not appear to test this distinction. Persona-disjoint evidence\textbf{Persona-disjoint evidence} only establishes that different personas are associated with different collections; it does not establish that the correct target response is personalized.

Prior Works and SOTA deployed LLMs already cover project-scoped reference resolution

The motivating scenerio is very close to the workspace/project abstraction used in systems such as ChatGPT, Claude projects: a project contains its own files, conversations and cotexts, and a question referring to that project is answered using corresponding project materials. For example, if I ask a question about project A, the system should naturally use Project A's files rather than files belonging to an unrelated project B. I view this as project/reference scoping. Unless experiments reveal something very interesting and fundamental regarding such deployed settings (like whether the new benchmark exposes why the current user-specific projects and the followup discussions are not trustworthy), I do not find there is any specific contribution of this benchmark.

Empirical results do not resolve the issue

The experimental findings revolve around:

  • memory systems performing worse than shared backbone
  • positioning of anchor matters
  • comnpression of evidence and its effect

and so on.

These findings do not demonstrate that personalization\textbf{personalization} is the source of difficulty. Therefore,

$ \text{interesting cross-source memory/RAG failure} \neq \text{evidence of a personalization failure}. $

Why this issue is Decision Critical

The issue is not a mere terminology or re-scoping issue that can be addressed via title or wording. Under the current construction, I do not see evidence that the benchmark evaluates personalization beyond persona/project-scoped reference resolution and evidence retrieval. Resolving this concern would therefore require substantial reframing around cross-source or project-scoped evidence integration, or additional evaluation where user-specific information genuinely changes the appropriate target response and where the claimed gap is demonstrated against systems that instantiate the project/workspace setting used to motivate the paper.

Questions for authors

I have no additional questions

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer sLVS

Response to Reviewer sLVS

Thank you for pinpointing this core conceptual distinction. Below we adopt your suggested reframing across the manuscript, compare flat and structured memory baselines within bounded persona workspaces across three backbones, and state our open-architecture scope.


Reply to W1

1. Task formulation: project-conditioned document-evidence resolution. Your GreenShapes example captures the exact structure of our benchmark: chat history identifies which project, study, or comparison is referenced, while the scientific ground-truth answer (3D PRINTING) remains invariant once that referent is resolved. Following your recommendation for substantial reframing, we revise the Abstract, Section 1 (Table 1), Section 2.1, Section 4, Section 6, Appendix A.1 (Table 5), and Appendix E to define the evaluated capability explicitly as project-conditioned document-evidence resolution and integration, removing claims that user state alters ground-truth answers to identical resolved queries and noting counterfactual preference-driven QA as future work.

2. Multi-backbone comparison within bounded persona workspaces. Each released persona context already forms a bounded research workspace of 4–28 paper blocks (196 blocks across 10 personas, 72k–168k tokens with chats in Appendix B.2). Across all n=266n = 266 questions on three backbones (Section 5.2, Appendix C.10, Figure 7 and Tables 22–24), structured entity-graph indexing surpasses flat workspace baselines on L2 across all three backbones and on L1 under newer backbones:

Persona workspace method (n=266n = 266)OAI-4o-mini L1 / L2 (%)DeepSeek-V4.1-Flash [3] L1 / L2 (%)Hy4 [4] L1 / L2 (%)
Full context (workspace reader)50.0 / 28.758.27 / 27.8257.89 / 28.20
BM25 (top-10 workspace chunks)49.6 / 29.256.77 / 31.9561.28 / 34.59
Cognee (hybrid graph + vector)47.3 / 30.969.17 / 28.9567.29 / 42.48
HippoRAG-v2 [1] (KG + PPR)46.1 / 32.061.65 / 43.6168.80 / 42.48
RAPTOR [2] (tree summarization)30.1 / 18.631.58 / 18.0531.58 / 17.67

OAI-4o-mini reports three-run means (Table 3), while DeepSeek-V4.1-Flash and Hy4 report complete single-run matrices (Appendix C.10, Tables 22–23).

Even inside a 4–28 paper workspace, flat reading (Full context), lexical retrieval (BM25), and recursive summarization (RAPTOR [2]) often miss study-specific qualifiers—across 3,974 method–query instances, L1 drops from 63.3% to 39.3% when normalized gold-answer strings are absent from generator inputs (Section 5.4)—whereas passage-preserving entity-graph stores (Cognee, HippoRAG-v2 [1]) achieve statistically significant gains over Full context on DeepSeek-V4.1-Flash and Hy4 (Holm pH=0.032p_H = 0.032 and 0.0370.037 on L1, pH<0.001p_H < 0.001 on L2).

3. Open-architecture scope. PerDocBench evaluates 16 open, reproducible memory architectures under controlled corpus and prompt protocols (Appendices C.10–C.13, Appendix E), complementing proprietary commercial workspace interfaces whose closed routing layers cannot be ablated.


References

[1] Gutiérrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (HippoRAG 2). arXiv:2502.14802, 2025.

[2] Sarthi et al. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. ICLR, 2024.

[3] DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017, 2026.

[4] Tencent. Tencent Launches Hy4, Joining the Ranks of the World's Leading Open-Source AI Models. Official release & public API: https://openrouter.ai/tencent/hy4, 2026.

Reviewer_sLVSOpenReview ↗

Sorry for not being able to revise my score

Thank you for the detailed response and for clarifying the revised positioning. I appreciate that the authors have acknowledged the distinction raised in my review. However, I am still not convinced that the response resolves my main concern, for two reasons.

1. The proposed reframing is substantial, rather than a clarification of the original problem.

My original concern was that the benchmark evaluates project/persona-scoped evidence resolution rather than personalization. The response largely agrees with this observation and proposes to reposition the work around project-conditioned document-evidence resolution.

I see this as a substantial change to the central problem statement, not simply a clarification of its terminology. Such a change would require reconsidering the motivation, related literature, contribution claims, and interpretation of the results under a different problem formulation. While I appreciate the authors' willingness to revise, I do not think that accepting a different problem positioning during rebuttal is, by itself, sufficient justification for moving my score from clear reject to weak reject. In my view, the revised framing would require a more complete reassessment of the work.

2. Under the revised framing, the main empirical findings appear to overlap considerably with established literature.

My second concern is what new scientific understanding the benchmark provides once the personalization claim is removed. Several of the observations emphasized in the response have already been studied in the long-context, RAG, compression, and LLM-memory literature.

Finding emphasized in the submission/rebuttalRelevant prior workFinding already established
Performance changes when the conversational anchor and document corpus are presented in different orders.Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2024)LLM performance can depend strongly on where relevant information appears within a long context. While the source-order experiment is different, the broader phenomenon of positional sensitivity is already established.
Several memory/retrieval systems perform worse than directly providing the available context to the backbone.Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (Li et al., 2024)When sufficient context capacity is available, direct long-context inference can outperform retrieval-augmented approaches, despite their computational advantages.
Compression and summarization can remove answer-bearing evidence and reduce answer accuracy.Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs (Cao et al., 2024)Loss of key information during compression can substantially degrade QA performance, motivating methods that explicitly preserve query-relevant evidence.
Preserving original evidence can be more reliable than repeatedly rewriting or summarizing memory.AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models (Shen et al., 2026)Repeated memory rewriting and summarization can dilute contextual details. The work separates retrieval anchors from preserved original context to support more reliable memory retrieval and generation.

I am not suggesting that these papers evaluate exactly the same benchmark setting. However, the underlying findings concerning positional sensitivity, retrieval versus full-context trade-offs, information loss through compression, and the importance of retaining original evidence are not new.

The additional multi-backbone experiments provide further evidence that these behaviors occur within the proposed benchmark. However, I do not see how they establish a substantially new empirical insight under the revised project-conditioned evidence-resolution setting, beyond demonstrating already recognized failure patterns in another evaluation environment.

Overall, the response acknowledges my original concern but addresses it primarily through a different problem framing. Under that framing, I remain unconvinced about the novelty of the central empirical findings. Therefore, my assessment remains unchanged, and I retain my original score.

Reviewer_MBn2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PerDocBench, a benchmark for evaluating personalized memory systems on cross-source evidence integration. Each of 10 scholar personas pairs a short research conversation (providing a routing cue via project codenames) with a persona-specific library of real papers (97 parsed PDFs) supplying the answer-bearing evidence. The benchmark contains 266 QA pairs across four question types, built via an LLM-driven pipeline with deterministic and semantic quality filters. The authors evaluate 16 memory methods under a shared OAI-4o-mini backbone and report a negative-utility finding: 15 of 16 methods score below the backbone on L1. A source-reordering ablation (anchor-first) yields a median L1 gain of 7.1%, and a distractor-rewrite stress test shows up to 63.6% of correct predictions flip under answer-preserving density rewrites.

Critical strengths

S1. Targets a real gap: unlike chat-only benchmarks where the answer fact appears in one compact utterance, PerDocBench separates the conversational routing cue from the answer-bearing document corpus. S2. The negative-utility finding is striking and non-obvious. 15 of 16 memory methods fall below their own backbone, and higher construction time correlates negatively with accuracy. S3. The quality-control pipeline is more rigorous than typical for benchmark papers, combining deterministic filters, a four-axis LLM semantic assessment, and a human validation study.

Critical weaknesses

W1. The benchmark is very small (266 QA pairs, 10 personas), and three-run standard deviations are comparable to the reported gaps, so the headline claims rest on limited statistical power. W2. The main comparison uses a single backbone and a shared interface that strips method-specific components (e.g., MemPalace KG variants, REMem graph walk, Self-RAG reflection tokens), likely biasing the comparison against memory methods. W3. The hackability claim is overstated: the distractor rewrite changes length, wording, and density simultaneously, and the answer-preservation check is performed by the same LLM family that generated the rewrite. W4. The anchor-first finding is confounded: the backbone itself gains 12 points under reordering, suggesting a general long-context position effect rather than a memory-specific insight.

Questions for authors

Q1. Given that standard deviations match the reported gaps, can you provide paired per-question significance tests for the claim that 15 of 16 methods fall below the backbone? Q2. Can you run the full 16-method comparison under at least one additional backbone to show the negative-utility pattern is not an OAI-4o-mini artifact? Q3. How much of the reported degradation is attributable to interface-induced reductions (MemPalace, REMem, Self-RAG) rather than the memory designs themselves? Q4. What fraction of the 266 questions can be answered correctly using only the anchor conversations, or only the document corpus? This would directly test cross-source necessity.

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer MBn2 (Part 1/2)

Response to Reviewer MBn2 — Part 1 of 2

Thank you for the rigorous questions on statistical testing, multi-backbone generality, interface fairness, and intervention controls. Below we report paired tests, two additional 17-configuration backbone matrices, component restorations, and ordering/source ablations across all n=266n = 266 questions.


Reply to W1 & Q1

We attach two-sided exact McNemar tests (Holm pHp_H) and 10,000-draw persona-cluster 95% bootstrap CIs to the paired backbone comparisons across all n=266n = 266 questions (Section 5.2, Appendix C.10, Figure 7 and Tables 22–24). Regarding Q1, Table 3's 15/16 summarizes three-run mean L1 rankings under OAI-4o-mini, whereas paired testing across all evaluated methods separates statistically significant differences from sampling variation:

Backbone (n=266n = 266, vs. Full context)Mean L1 < FullSig. Lower (L1 / L2)Non-sig. (L1 / L2)Sig. Higher (L1 / L2)
OAI-4o-mini (Table 24, Holm-32)15/16 (3-run)6 / 39 / 120 / 0
DeepSeek-V4.1-Flash [5] (Table 22, Holm-64)13/168 / 37 / 121 / 1
Hy4 [6] (Table 23, Holm-64)10/16 (1 tie)5 / 310 / 111 / 2

Significance at Holm pH<0.05p_H < 0.05. OAI-4o-mini tests all 15 archived single-run logs (BM25's two slots retained in Holm-32).

Under OAI-4o-mini, 6 methods (Contriever, MemoRAG [4], RAPTOR, Self-RAG [3], SimpleMem, GAM) score significantly below Full context on L1 (pH≤0.0013p_H \le 0.0013) with negative persona-cluster 95% CIs, while top lexical and embedding retrievers (OAI-Emb-3-Small/Large, Qwen3-Embedding-4B) lie within sampling variation (Section 5.2 is calibrated accordingly). On DeepSeek-V4.1-Flash [5] and Hy4 [6], significant positive gains emerge in entity-graph stores (Cognee, HippoRAG-v2 [1]), detailed below.


Reply to W2 & Q2

Complete 17-configuration evaluations (n=266n = 266) on DeepSeek-V4.1-Flash [5] and Hy4 [6] show that PerDocBench separates positive and negative memory utility (Section 5.2, Appendix C.10, Figure 7 and Tables 22–23):

Method (n=266n = 266)DeepSeek-V4.1-Flash L1 / L2 (%)DS Δ\DeltaL1 / Δ\DeltaL2 (%)Hy4 L1 / L2 (%)Hy4 Δ\DeltaL1 / Δ\DeltaL2 (%)
Full context58.27 / 27.82—57.89 / 28.20—
BM2556.77 / 31.95−1.50-1.50 / +4.14+4.1461.28 / 34.59+3.38+3.38 / +6.39+6.39
Cognee69.17 / 28.95+10.90+10.90 / +1.13+1.1367.29 / 42.48+9.40+9.40 / +14.29+14.29
HippoRAG-v2 [1]61.65 / 43.61+3.38+3.38 / +15.79+15.7968.80 / 42.48+10.90+10.90 / +14.29+14.29
RAPTOR31.58 / 18.05−26.69-26.69 / −9.77-9.7731.58 / 17.67−26.32-26.32 / −10.53-10.53
GAM34.59 / 10.15−23.68-23.68 / −17.67-17.6746.99 / 25.56−10.90-10.90 / −2.63-2.63

Bold Δ\Delta entries denote Holm pH<0.05p_H < 0.05 across 64 two-backbone tests (full 17-method matrices in Tables 22–23).

Across all 16 methods, 13/16 trail Full context on L1 under DeepSeek-V4.1-Flash and 10/16 under Hy4 (one tie, five gains): lossy summarization consistently degrades fact recovery, whereas passage-preserving entity-graph stores achieve significant gains (Cognee pH=0.032p_H = 0.032 on DS L1, HippoRAG-v2 pH=0.037p_H = 0.037 on Hy4 L1, pH<0.001p_H < 0.001 on L2). Tables 22–23 report all 17 configurations, including SimpleMem's structured-JSON builder sensitivity on Hy4 across all 266 queries.


Reply to W2 & Q3

Across all n=266n = 266 questions, we restore upstream traversal and reranking over identical 10-persona memory banks under DeepSeek-V4.1-Flash [5] and evaluate standalone fine-tuned Self-RAG-7B [3] weights with reflection tokens (Section 5.2, Appendix C.11):

Configuration (n=266n = 266)Adapted L1 / L2 (%)Restored / Native L1 / L2 (%)Δ\Delta L1 / L2 (%)
Full context (DeepSeek-V4.1-Flash)64.29 / 33.08—Reference
REMem [2] (+ PPR & DSPy rerank)51.50 / 25.9462.78 / 28.95+11.28+11.28 / +3.01+3.01
MemPalace (+ Room & KG routing)18.42 / 10.1527.82 / 15.04+9.40+9.40 / +4.89+4.89
Self-RAG-7B [3] (standalone 7B)—16.92 / 10.53Fine-tuned 7B weights

Restoring PPR traversal + DSPy reranking in REMem [2] (p<0.001p < 0.001) and room + KG routing in MemPalace (p<0.001p < 0.001 on L1, p=0.015p = 0.015 on L2) substantially improves within-family accuracy, yet neither surpasses matched Full context, and standalone Self-RAG-7B remains below long-context readers. Interface standardization thus accounts for part of within-method loss while preserving our core comparison against Full context.

AuthorsOpenReview ↗

Official Response to Reviewer MBn2 (Part 2/2)

Response to Reviewer MBn2 — Part 2 of 2


Reply to W3

We agree that the Section 3 chat-rewrite probe jointly modifies wording, length, and local turn structure under an LLM self-check. Accordingly, we revise Section 1, Section 3 (Table 2), and Appendix C.1 to present these results strictly as an exploratory motivating observation of presentation sensitivity in conversational benchmarks, removing "benchmark hackability" terminology and isolated density-causality claims while grounding all primary conclusions in PerDocBench's controlled evaluations.


Reply to W4

We agree that prompt-position effects contribute to Table 15's single-run reordering shift. To separate index construction (Write) from reader prompt ordering (Read), we run a 2×22 \times 2 factorial for BM25 and MemoRAG [4] alongside a matched Full context position control across all n=266n = 266 questions on DeepSeek-V4.1-Flash [5] (Section 5.3, Appendix C.13, Tables 26–27):

Ordering condition (n=266n = 266)BM25 L1 / L2 (%)MemoRAG [4] L1 / L2 (%)
Write corpus / Read corpus63.91 / 32.3360.90 / 28.95
Write corpus / Read anchor66.54 / 34.5959.02 / 29.32
Write anchor / Read corpus64.66 / 29.3260.90 / 29.32
Write anchor / Read anchor66.54 / 32.7160.15 / 29.70

Matched Full context on the same 266 questions: Corpus-first 66.17 / 30.08%, Anchor-first 65.41 / 34.21% (Appendix C.13).

Varying Write order changes retrieved passages in 77.4% of MemoRAG pairs versus 0% for BM25 (exact indexing invariance), yet every Full context-adjusted read-order contrast (DwD_w) has a 95% persona-cluster CI crossing zero across both methods (Table 27). We therefore revise Section 1, Section 5.3, and Section 6 to frame source ordering as a stage diagnostic without claiming a universal anchor-first advantage.


Reply to Q4

Evaluating all n=266n = 266 questions under DeepSeek-V4.1-Flash [5] with fixed queries, gold answers, and judging (Section 5.3, Appendix C.12, Table 25) directly quantifies single-source answerability:

Input channel (n=266n = 266, DeepSeek-V4.1-Flash)L1 (%)L2 (%)
No context21.0513.16
Anchor conversations only7.524.51
Document corpus only66.9229.32
Full context (Anchors + Documents)64.2933.08

Factual answers reside predominantly in the paper library, and persona-cluster 95% CIs for Full context minus Document corpus only cross zero on both metrics (Table 25). Combining both sources solves 16 L1 and 21 strict L2 items where both isolated channels fail—resolving target project scope without unsupported detail—which motivates our calibrated formulation of project-conditioned document-evidence resolution and integration across Sections 1, 4, and 5.3.


References

[1] Gutiérrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (HippoRAG 2). arXiv:2502.14802, 2025.

[2] Shu et al. REMem: Reasoning with Episodic Memory in Language Agent. arXiv:2602.13530, 2026.

[3] Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR, 2024.

[4] Qian et al. MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation. WWW, 2025.

[5] DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017, 2026.

[6] Tencent. Tencent Launches Hy4, Joining the Ranks of the World's Leading Open-Source AI Models. Official release & public API: https://openrouter.ai/tencent/hy4, 2026.

Reviewer_KKgoOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes PerDocBench, a benchmark for evaluating personalized memory systems' ability to integrate conversational cues with evidence from scholarly document collections. Experiments show that existing methods underperform the shared LLM backbone, while placing conversations before documents improves performance.

Critical strengths
  • The benchmark development process was well-structured, and existing memory-related methods were extensively analyzed through experimentation.
Critical weaknesses
  • Lack of differentiation from existing benchmarks: The benchmark proposed in this paper is essentially a personalized academic QA benchmark. In other words, solving this benchmark requires identifying the user's needs for a given query based on their history, retrieving relevant information from the document collection, and generating an answer. This is not substantially different from existing personalized QA benchmarks such as LaMP-QA and BESPOKE. In fact, compared to these benchmarks, the proposed benchmark has a more limited domain and evaluation scope (restricted to the academic domain and only evaluating "correctness").
    • Furthermore, compared to other benchmarks that consider multiple sources, Tables 1 and 5 claim that the proposed benchmark differs from others in terms of "X-source" and "Q-Type," but the supporting evidence is insufficient. After examining these benchmarks, I believe that other benchmarks also require more than simple factual lookup and involve questions that demand reasoning across source types. Therefore, additional explanations should have been provided, at least in the Appendix.
  • Lack of connection between the motivation in Section 3, the proposed benchmark, and the experimental section: Section 3 investigates evidence density and demonstrates that performance is sensitive to evidence density. However, based on these findings, this paper concludes that cross-source evaluation is necessary.
    • As a result, the necessity of cross-source evaluation is not sufficiently justified. In fact, as shown in Section 3, evidence density can be sufficiently low even in single-source settings. If all chat histories were structured as shown in Section 3, this would still constitute an environment with low evidence density, even in a single-source setting. To demonstrate that cross-source settings are more challenging, I believe that identifying and experimentally demonstrating challenges unique to cross-source settings, which are clearly distinct from those in single-source settings, rather than focusing on evidence density, would better motivate the necessity of the proposed benchmark.
    • Furthermore, in the experimental section, the analysis of why existing methods perform poorly focuses only on their functional characteristics, without investigating why they perform poorly specifically in multi-source settings. As a result, the challenges of multi-source settings emphasized by the proposed benchmark are not directly addressed.
Questions for authors
  • Could you elaborate on how the proposed benchmark differs from existing benchmarks?
  • Could you explain the challenges specific to cross-source settings in more detail, particularly how they differ from cases where evidence density is low in single-source settings?
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer KKgo

Response to Reviewer KKgo

Thank you for the constructive comments on benchmark positioning and cross-source motivation. Below we clarify our comparison with prior benchmarks, report 266-question single-source ablations, and connect them to why memory pipelines fail.


Reply to W1 & Q1

PerDocBench complements prior personalized QA benchmarks by evaluating project-cue resolution and verifiable fact recovery across multi-paper scholarly libraries over four question types (Technical Detail, Section Retrieval, Document Browsing, and Multihop Reasoning):

BenchmarkEvidence sources & scaleHistory role & evaluated target
LaMP-QA [1]Past user questions & descriptions (3 domains)Personalized information-need context for rubric-graded long-form QA
BESPOKE [2]30 users, chat + search logs (150 queries)Intent context for search-augmented rubric adaptation
ATM-Bench [3]1 user, ~12k emails, photos, videos (1,038 QA)Referential personal archive (~30% multi-evidence QA)
PerDocBench10 personas, anchor chats + 196 paper blocks (266 QA)Project/study scoping for verifiable document-fact recovery

We agree that prior multi-source benchmarks involve cross-source reasoning and diverse queries. Appendix A.1 (Table 5) incorporates LaMP-QA [1], while Section 1 (Table 1) and Section 2.1 clarify X-source and Q-type criteria by contrasting concrete evidence carriers and history roles. Focusing on scholarly PDFs with L1 fact recovery and strict L2 faithfulness provides inspectable ground-truth spans that isolate memory-induced qualifier loss from open-ended rubric alignment.


Reply to W2 & Q2

1. Single-source ablation and role of conversational cues. Across all n=266n = 266 questions under DeepSeek-V4.1-Flash [4] (Section 5.3, Appendix C.12, Table 25), we evaluate isolated and combined input channels:

Input channel (n=266n = 266, DeepSeek-V4.1-Flash)L1 (%)L2 (%)
No context21.0513.16
Anchor conversations only7.524.51
Document corpus only66.9229.32
Full context64.2933.08

Documents supply the primary factual evidence (persona-cluster 95% CIs for Full context minus Document corpus only cross zero on both metrics), while anchor chats disambiguate target project scope and solve 16 L1 and 21 strict L2 items where both single sources fail. Sections 1, 3, 4, and 6 therefore calibrate the task as project-conditioned document-evidence resolution and integration.

2. Presentation sensitivity vs. why memory methods fail. Section 3 frames the chat-rewrite probe solely as an exploratory observation of presentation sensitivity, distinct from cross-source necessity. On PerDocBench (Sections 5.2 and 5.4), stage-level case analyses show lossy pipelines (RAPTOR, GAM) altering or omitting study-specific qualifiers during compression, while across 3,974 instances L1 drops from 63.3% to 39.3% when normalized gold-answer strings are absent from generator inputs. Conversely, passage-preserving entity-graph stores (Cognee, HippoRAG-v2) link project cues back to verbatim passages and surpass Full context by +10.90% L1 on DeepSeek-V4.1-Flash [4] and Hy4 [5] (Holm pH=0.032p_H = 0.032 and 0.0370.037, Appendix C.10, Tables 22–23).


References

[1] Salemi and Zamani. LaMP-QA: A Benchmark for Personalized Long-Form Question Answering. EMNLP, 2025.

[2] Kim et al. BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback. arXiv:2509.21106, 2025.

[3] Mei et al. According to Me: Long-Term Personalized Referential Memory QA (ATM-Bench). arXiv:2603.01990, 2026.

[4] DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017, 2026.

[5] Tencent. Tencent Launches Hy4, Joining the Ranks of the World's Leading Open-Source AI Models. Official release & public API: https://openrouter.ai/tencent/hy4, 2026.

Thank you for your detailed response and the additional experiments. I believe that my previous concerns have been addressed to some extent, and I will raise my score accordingly.

However, I still have one significant concern. In my view, the paper's central claims appear to have changed substantially from those presented in the original submission.

Specifically, I feel that the rebuttal and additional experiments go beyond simply clarifying or strengthening the original claims. Rather, the motivation for the benchmark and the main claims about what it aims to evaluate seem to have shifted considerably throughout the discussion.

While I appreciate the authors' efforts to address my concerns, I believe these changes raise questions about the overall coherence and completeness of the original submission. In particular, the revised motivation and claims would need to be consistently reflected throughout the paper and supported by the corresponding analyses.

For this reason, although I acknowledge the improvements made during the rebuttal and will increase my score accordingly, I do not feel that I can justify raising it to an acceptance-level score.

AuthorsOpenReview ↗

Response to Reviewer KKgo

Thank you for your thoughtful engagement throughout the discussion and for updating your assessment.

We respectfully note that our core benchmark contribution and task design remain unchanged from the original submission, which already framed PerDocBench around connecting conversational project cues to answer-bearing facts in persona-specific paper libraries and motivating evidence-preserving memory construction in Section 1. The refined phrasing of project-conditioned document-evidence resolution and integration and the supplementary ablations were introduced to address reviewer questions, rule out confounding explanations, and state scope boundaries with greater precision. All revised formulations and core empirical analyses are coherently integrated throughout the uploaded revised manuscript, highlighted across the Abstract, Sections 1–6, and the Appendices.

Thank you again for your time and constructive feedback, which have helped make the paper clearer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up Response to Reviewer KKgo

Thank you for the insightful follow-up and constructive oracle-ablation suggestion, now completed across the full benchmark.


Reply to Additional Question 1

Yes, the BrowseComp vs. BrowseComp-Plus analogy captures a central design distinction. Freezing a span-verifiable workspace of scholarly PDFs and multi-session chats with two-tier L1 core-fact and strict L2 qualifier grading removes live-search variance and subjective style rubrics as in LaMP-QA and BESPOKE, while extending beyond single-user lifelogs such as ATM-Bench. This controlled design isolates personalized workspace scoping and faithful evidence integration.


Reply to Additional Question 2

1. Why chat-only accuracy falls below No context. With No context, the reader answers from general scientific priors and rarely abstains. Supplying personal chats without the referenced papers converts 60.7% of those No context L1 hits into non-answer responses under raw anchor chats and 50.0% under question-relevant research chats, with model outputs explicitly noting that the target study details are missing from memory. Lower chat-only accuracy is thus consistent with grounded restraint when referenced papers are absent.

2. Full-benchmark Oracle ablation. Across the full benchmark under the identical DeepSeek-V4.1-Flash reader, we pair blindly selected question-relevant research chats (OconvO_{\text{conv}}, chosen without documents or gold answers) with adjudicated supporting paper spans (OdocO_{\text{doc}}):

Context condition (DeepSeek-V4.1-Flash)Median tokensL1 (%)Strict L2 (%)Non-answer (%)
No context021.0513.1627.82
Anchor conversations only49.5k7.524.5178.57
Conversation oracle only (OconvO_{\text{conv}})0.6k9.776.3971.80
Document corpus only47.8k66.9229.329.40
Full context108.3k64.2933.0811.28
Document corpus + Conversation oracle (OconvO_{\text{conv}})48.4k66.9235.7111.65
Joint oracle (Odoc+OconvO_{\text{doc}} + O_{\text{conv}})1.4k72.1849.6214.66

Raw conversation logs interleave casual personal-life dialogue with research discussions. Pairing the document corpus with OconvO_{\text{conv}} eliminates the raw-chat L1 drop and lifts strict L2 faithfulness by +6.39% over Document corpus only. Conversational cues carry an annotated scoping, constraint, or answer role across 46.2% of the benchmark. On complementary queries requiring both project cues and paper facts, adding OconvO_{\text{conv}} to the document corpus lifts L1 from 48.15% to 62.96% and more than doubles strict L2 from 14.81% to 33.33%. Finally, Joint oracle climbs +7.89% in L1 and +16.54% in strict L2 above Full context at a fraction of the context budget, while the remaining headroom under this lightweight flash reader shows that PerDocBench requires genuine multi-hop synthesis and strict qualifier faithfulness beyond extractive span lookup.

3. Why this evaluates personalized context utilization beyond generic RAG. Unlike standalone document RAG with self-contained queries, PerDocBench requires systems to distill sparse project cues from personal history and bind them to paper evidence without query dilution. On document-grounded queries, naively concatenating OconvO_{\text{conv}} into BM25 lowers target-paper Hit@1 from 65.34% to 43.03% as conversational phrasing dilutes technical terms. This aligns with our main benchmark in Appendix C.10, where flat lexical RAG and lossy summarizers struggle while passage-preserving entity-graph memory such as Cognee and HippoRAG-v2 achieves substantial gains over Full context.

AuthorsOpenReview ↗

Follow-up Response

Thank you for carefully reviewing the new experiments. Below we concisely address both points.


1. Cross-backbone consistency and benchmark contribution

Lossy memory pipelines degrade accuracy across all three backbones, while passage-preserving entity-graph stores consistently lead.

  • Consistent failure of lossy pipelines. Holm-significant L1 drops below Full context span 8 methods on DeepSeek-V4.1-Flash, 6 on OAI-4o-mini, and 5 on Hy4 in Appendix C.10, Tables 22–24, with RAPTOR, SimpleMem, and Contriever significantly below Full context on all three backbones.
  • Interface fairness and consistent winners. Restoring upstream graph traversal and reranking in Appendix C.11 still leaves REMem and MemPalace below Full context, as does standalone Self-RAG-7B. Under the same standardized interface, Cognee and HippoRAG-v2 claim first place on both L1 and strict L2 across every backbone, achieving Holm-significant gains over Full context on DeepSeek-V4.1-Flash (Cognee L1, HippoRAG-v2 L2) and Hy4 (Cognee L2, HippoRAG-v2 L1 and L2).
  • Benchmark value. Separating lossy compression from passage-preserving entity-graph memory across backbones is a core diagnostic strength, as reflected in our calibrated Abstract and Sections 5.2 and 6.

2. Why Document corpus only slightly exceeds raw Full context on L1

Adding chats shifts the reader from broad topic matching to project-conditioned verification, and focused project chats (OconvO_{\text{conv}}) eliminate the L1 dip while lifting strict L2 faithfulness. Extending Table 25 in Appendix C.12 (Reply to Q4), we evaluate question-relevant research chats (OconvO_{\text{conv}}) and reference supporting paper spans (OdocO_{\text{doc}}) under DeepSeek-V4.1-Flash:

Context condition (DeepSeek-V4.1-Flash)Median tokensL1 (%)Strict L2 (%)
Document corpus only (Table 25)47.8k66.9229.32
Full context (Table 25, papers + raw chats)108.3k64.2933.08
Document corpus + Conversation oracle (OconvO_{\text{conv}})48.4k66.9235.71
Joint oracle (Odoc+OconvO_{\text{doc}} + O_{\text{conv}})1.4k72.1849.62
  • Why a slight L1 dip under raw chats is expected. Without chats, Document corpus only sees no private project codenames and falls back to broad topic matching across the scholar's papers, where 76.0% of the lenient L1 hits lost under Full context already fail strict L2. Once raw chats are added, the reader checks project alignment and cautiously abstains when casual dialogue obscures a paper link, while upgrading aligned items into faithful strict L2 answers.
  • Focused project cues recover L1 and boost strict L2. Pairing documents with OconvO_{\text{conv}} removes raw-chat noise, eliminating the L1 dip and raising strict L2 by +6.39% over Document corpus only. On complementary queries drawing on both project cues and paper facts, adding OconvO_{\text{conv}} lifts strict L2 from 14.81% to 33.33%, and Joint oracle reaches +16.54% in strict L2 above Full context.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.