- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates privacy leakage during the consolidation of private conversations into shared agent memory. The central concern is that private information may be incorporated into team-visible profiles or digests even when query-time access controls are correctly enforced. The authors construct a synthetic workplace benchmark and use matched source counterfactuals to examine whether private facts propagate into shared artifacts. They further evaluate exposure under different memory templates, privacy instructions, and ordinary user queries. Finally, the paper compares several configurations for authorized information sharing, including native agent isolation and an owner-confirmed release interface. The results demonstrate substantial privacy exposure in configurations without structural source binding, while audience isolation and trusted source filtering substantially reduce the observed leakage.
Critical strengths
- The paper examines an important privacy boundary in multi-user agent systems: private information can enter shared artifacts even when subsequent access controls operate correctly. The non-injected threat model is particularly relevant because leakage can occur through ordinary collaborative interactions.
- The authors conduct matched source counterfactual experiments, extend the benchmark to 48 personas, examine multiple writer and judge models, and include independent human reannotation. The LOFO controls provide useful evidence that observed disclosures depend on the private-source content.
- The Mem0, LettaBot, and OpenClaw case studies complement the synthetic benchmark and demonstrate that the audited configurations are not purely hypothetical. The authorized-sharing experiments also highlight the practical
Critical weaknesses
- Limited novelty beyond existing contextual-integrity research. The central privacy risk substantially overlaps with prior work, particularly PiSAs and Collaborative Memory. While the paper adds source counterfactuals and software case studies, it remains unclear what fundamentally new failure mode or diagnostic capability it establishes beyond existing studies of cross-user privacy leakage and shared agent memory.
- Confounded primary comparison and unclear mechanism. The Profile–Digest comparison simultaneously changes organizing goals, field semantics, and person-slot counts (4 vs. 2). The borderline facts also closely match Profile's scheduling-related fields. Although additional factorial controls are provided, they do not consistently isolate the contributing factors. Thus, the primary result demonstrates a difference between two specific templates rather than a generalizable insight into memory organization.
- Unclear technical contribution of write-path auditing. LOFO establishes that leakage depends on private-source content but does not identify where privacy restrictions fail. The main benchmark collapses memory writing and consolidation into a single summarization step, while the real-system case studies examine different stages without a unified evaluation. Consequently, the proposed audit appears difficult to distinguish from conventional privacy-sensitive summarization and source-ablation testing.
- Unproven benefit of Confirmed Release. Native Isolation already achieves no detected privacy disclosure on complete-evidence trajectories. Confirmed Release improves joint success from 26/32 to 30/32, but the difference is not statistically significant and relies on an idealized owner. The paper does not demonstrate what additional failure mode this mechanism prevents compared with native isolation.
- Sensitivity of the primary findings to evaluation methodology. The Profile–Digest contrast varies considerably across evaluators. Independent human reannotation reduces the pooled difference to 0.094, and the extension cohort's confidence interval includes zero. Documented confusion between scheduling constraints and private reasons further raises concerns about annotation reliability and the robustness of the primary conclusion.
Questions for authors
- What new failure mode or diagnostic capability does write-path auditing offer beyond existing contextual-integrity research (e.g., PiSAs)? Given that the main benchmark collapses writing and consolidation, how does LOFO provide stage-specific insights beyond conventional source-ablation tests?
- How can the Profile–Digest difference be interpreted when multiple factors (field semantics, organizing goals, and slot counts) change simultaneously? Moreover, how do the authors reconcile the substantially different estimates from automated and human evaluations?
- What additional security benefit does Confirmed Release provide over Native Isolation, given that both show no detected leakage on complete-evidence trajectories and the improvement in joint success is not statistically significant?
Your main point is fair: the spillage phenomenon is not new, and Confirmed Release does not add a proven security benefit. We have narrowed the claims in the revised paper.
PiSAs, Collaborative Memory, and LOFO. PiSAs already studies non-adversarial spillage across shared memory and derived notes, including availability versus private reasons and paraphrase-tolerant evaluation. Collaborative Memory already provides private/shared tiers, immutable provenance, and read/write policies. We do not claim either the spillage phenomenon or provenance-based isolation as a new discovery.
You are also right that LOFO is a source-ablation control, not a stage-specific diagnostic. Trellis merges memory writing and consolidation into one transcript-to-artifact step, and the Mem0, LettaBot, and OpenClaw cases examine different system boundaries rather than one unified pipeline. LOFO therefore supports the private source's contribution to disclosure. Fixed-transcript controls also show that assistant replies can retain private facts. It does not show which internal stage failed.
Within that narrower scope, our increment is the combination of matched source counterfactuals, separate measurement of artifact inscription and recovery by an independent artifact-only reader, annotation error analysis, and executable runtime configuration comparisons.
Profile–Digest. The comparison bundles organizing goal, field titles, and person-slot count (4 vs. 2), and all borderline facts are scheduling-related. Our factorial and capacity results are not stable across judge families. The revised paper therefore describes a difference between these two concrete templates on these scheduling assets. It does not claim a general person-centered/event-centered design rule.
Annotation. On the 48-persona matrix, the pooled Profile-minus-Digest contrast is 0.229 under the automated judge, 0.260 under model-assisted adjudication, and 0.094 under returned independent-human labels. On the 32-persona extension, the returned-human estimate is 0.055 (95% CI
[-0.008, 0.117]), which includes zero and does not independently establish the contrast. The targeted review gives 0.255, but 174 of the 181 reviewed items were returned-positive by construction, so it cannot estimate residual error across a representative positive-and-negative sample.The substantive disagreement is whether an artifact that records only an availability constraint also discloses its omitted private reason. We report the protocols side by side rather than choosing the largest estimate as the preferred one.
Confirmed Release. We agree that it demonstrates no additional security benefit over Native Isolation. Native already achieves zero detected disclosure on all 29 complete-evidence runs. Confirmed Release's 30/32 versus 26/32 joint-success difference is not statistically significant and relies on an idealized deterministic owner.
Across the 128 runtime trajectories, 115 have complete evidence; the other 13 are 3 workflow failures, 8 generation timeouts, and 2 unjudged runs, none treated as privacy-safe. In the revised paper, Native Isolation is the primary effective baseline, and Confirmed Release is retained only as an executable configuration comparison.