- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MemJoin, a benchmark of synthetic cases where multiple sessions jointly reveal a sensitive relation and no individual session establishes. It evaluates six memory frameworks with a fixed reader, separates attribute availability from relation disclosure and tests privacy interventions. Retrieved-memory disclosure rates are from to .
Critical strengths
-
Well-controlled benchmark construction. Full-history, isolated-session and broken-link validations helps establish the target relation requires combining evidence across sessions. This isolate a important privacy failure mode that evaluation of explicitly supplied secrets could miss.
-
Useful diagnostic comparisons. Evaluating actual memory pipelines alongside full-history, and all-memory controls distinguishes storage, and retrieval effects. The cue-removal experiment further supports the conclusion that retaining target attributes alone don't determine disclosure.
Critical weaknesses
-
Limited validation in evaluation. All experiment uses one 9B reader in the non-thinking mode with task completion of only across retrieved-memory systems and under full history. Answer judgments are entirely automated without reported human agreement measurements. These limitation makes it difficult to assess if the observed trade-offs generalize to capable assistants. Moreover, lexical availability might miss valid paraphrases, weakening interpretations which attribute lower disclosure to lost information.
-
Utility-preservation claim is overstated. For cases, completion criteria explicitly prohibit unrelated personal details, entangling privacy with utility. Appendix F.2 states that output filtering’s entire TCR gain comes from these cases. It means completion on the other 204 cases falls from to . Cue removal also lowers helpfulness and roughly doubles refusals. Therefore, stable aggregate TCR doesn't establish that disclosure is reduced 'without costing the task'.
Questions for authors
-
Could you provide human validation and inter-annotator agreement for disclosure, task completion and helpfulness, including examples explaining low full-history TCR?
-
Does the principal finding persist with a stronger reader and reasoning enabled?
-
How sensitive is availability and conditional-disclosure conclusion to a semantic coverage check which recognizes paraphrases and aliases?
-
Could you report privacy-independent utility results separately for the content-based tasks, including uncertainty for M3’s helpfulness changes, and clarify which evidence supports utility preservation?
-
[1] directly addresses CI reasoning and privacy–utility tradeoffs through reasoning-based and RL-based mitigations. How could CI-focused post-training mitigate cross-session disclosure?
[1] Contextual Integrity in LLMs via Reasoning and Reinforcement Learning, NeurIPS 2025.
5 comments
- Content-task utility and M3
Our prespecified checked-value-retained comparison has 207 complete pairs per input: disclosed cases fall from 53 to 18 on Itemized history, 34 to 16 on A-Mem, and 31 to 4 on MemGPT (Appendix F.1). This retains evidence for the connecting-cue diagnostic when all checked target values remain visible. It does not establish cost-free task utility. We will remove “without costing the task” and report the 204 content-based tasks separately from the 84 privacy-constrained criteria. For output filtering, not M3, the mean content-task TCR across the six frameworks falls from 13.6% to 12.0% (Appendix F.2); M3 also lowers helpfulness and increases refusal. Content-task completion, helpfulness and refusal changes, including M3 uncertainty, are reported in Section 3 of our General Official Comment. M3 Help. changes (95% CIs) are −0.149 [−0.260, −0.042] for Itemized history, −0.136 [−0.248, −0.023] for A-Mem, and −0.147 [−0.265, −0.029] for MemGPT. We will describe the observed tradeoff rather than infer utility preservation from aggregate TCR or an interval spanning zero. We have added a supplementary utility analysis of the 204 content-based tasks; the results are shown in Section 3 of our General Official Comment.
- CI-focused post-training
Thank you for highlighting CI-focused post-training in Lan et al. (NeurIPS2025). We evaluated the released Qwen2.5-7B-Instruct-CI checkpoint on MemJoin. C0 uses Qwen2.5-7B-Instruct with the original prompt. C1 uses the same base weights with CI-CoT, and C2 uses the CI-trained weights with the same CI-CoT. We froze the original memory inputs across six frameworks and all 14 views, evaluating 288 cases with three trials per setting. We used the original LLM judges to assess disclosure, appropriateness, completion, and helpfulness under separate criteria. We used two NVIDIA H200 GPUs for the reader evaluation, with an estimated total of approximately 5 GPU-hours.
Under Retrieved memory, mean DR across the six frameworks decreases from 4.34% (C0) to 1.56% (C1) and 0.75% (C2). C2 still produces complete-relation disclosure in all six frameworks: DR ranges from 0.35% to 1.04% with Retrieved memory and from 0.69% to 2.43% with All memory, with 4.17% under Full history and 0% under Task only. Under Retrieved memory, mean TCR on the 204 content-based tasks falls from 9.07% (C1) to 3.76% (C2). These outcomes extend MemJoin's observed cross-session disclosure risk to the tested CI-trained reader and record mitigation together with a task-utility cost in this view. We will add the complete comparison and protocol to Appendix F.3 (CI-focused post-training), with a brief summary in Section 4.4 (RQ3). The results are shown in Section 1 of our General Official Comment.
References
Guangchen (Eric) Lan, Huseyin A. Inan, Sahar Abdelnabi, Janardhan Kulkarni, Lukas Wutschitz, Reza Shokri, Christopher Brinton, and Robert Sim. Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. Advances in Neural Information Processing Systems 38 (NeurIPS2025), 104355–104391. DOI: 10.52202/085713-3485.
We have uploaded the revised manuscript PDF. We welcome any further feedback and would be happy to discuss your questions, provide further clarification, and address any remaining concerns. We apologize for the unclear presentation of the LLM-agreement results cited in point 1 of our earlier response. Item 4 of our General Official Comment now reports the proportion of evaluations in which at least two of the three LLM judges agree, matching Appendix C.1.
Thank you for your careful review and constructive suggestions. In response to the reviewers' requests, we completed the supplementary evaluations presented in our General Official Comment. Below, we respond to your comments point by point.
We have completed a supplementary reader evaluation across model families and sizes. Below, we separate reader coverage from answer-judge validation and address the privacy–utility interpretation and measurement concerns.
Point-by-point Responses
Thank you for this suggestion. We have not conducted independent human annotation due to staffing constraints. Section 4 of our General Official Comment reports LLM agreement for disclosure, completion and helpfulness, which does not replace human validation. Selected Full history answers omit departure times or travel steps, leave rescheduling dates and times as placeholders, or append identifiers prohibited by the completion criteria. Some criteria require information absent from the history, so the 18.1% TCR must be read against these criteria and the available task information. We will report judge agreement in Appendix C.1 and the criteria and examples in Appendix C.3, with a discussion in Section 4.2.
Thank you for raising this question. We keep Qwen3.5-9B in non-thinking mode in the main comparison to hold the answer-generating model constant while comparing memory inputs. We have added six readers, giving seven models from Qwen, Gemma and Llama (3B–31B). The five Qwen/Gemma readers have thinking Off/On comparisons; the two Llama readers use native Instruct. We reuse the frozen writer, memory stores and retrieval inputs of all six frameworks, with the original 288 cases, three trials and scoring rules. Retrieved DR is 0.7%–20.1%, nonzero in every framework across all 12 settings. With thinking enabled, DR is 4.9%–20.1% for Qwen3.8-27B and 1.7%–6.3% for Gemma4-31B; Qwen3.8-27B TCR is 17.7%–23.6%. These results support our central finding that memory reuse can disclose sensitive cross-session relations across reader families, including larger thinking-enabled models. We will present the key DR/TCR results in Section 4.2 and complete the reporting in a new Appendix E.4 with Help., Ref., All-memory results and the full evaluation protocol. The isolated-session and broken-link support checks remain those of the original reader. Results are shown in Section 2 of our General Official Comment.
Thank you for raising this question. We use string matching against the accepted forms because the target values in MemJoin are fixed at case construction. Applying the same lexical availability check across memory views provides a reproducible reference for analysing relation disclosure and makes the distinction between attribute availability and relation disclosure explicit. Compared with relying entirely on an LLM for the availability check, string matching is easier to audit: each decision can be verified against the historical input and does not depend on a judge model, prompt, or model version. We therefore retain this lexical availability check to provide a consistent and traceable basis for comparing disclosure across frameworks.
To address your concern, we selected 48 fixed cases, two from each of 24 families. We compared the original lexical availability check with a semantic coverage check using gemini-3.6-flash across all 13 memory and history views. The historical inputs, reader answers, and A4 labels were kept unchanged. We then recomputed Avail. and Conditional A4. No input changed coverage classification in this sample, and both metrics differed by 0.00 percentage points in every view. Spot-checks found no substantive discrepancies in the inspected examples. Thus, switching to semantic coverage did not change these two metrics within the tested sample. We will report the results as a figure in Appendix E.5 and publicly release the complete benchmark, evaluation prompts and code, and supplementary results.