acceptodds
Under review as a conference paper at ICLR 2027

PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems

23 views

Abstract

As LLM agents evolve from single-user assistants into shared organizational infrastructure, new privacy risks emerge: inappropriate information may not only be exposed through outputs for external recipients, but also internally across users through inter-agent messages, shared memory and user-agent interactions. These data spillage risks are not captured by existing privacy benchmarks grounded in contextual integrity (CI) as they focus primarily on either single-user settings or interactions between independently owned agents. We introduce PiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks with dual CI annotations: whether an information is appropriate for the task, and which users may legitimately access it. This enables direct measurement of cross-user spillage across agentic system components and interfaces, such as outputs, inter-agent communication, and memory. PiSAs is system-agnostic and supports evaluation across different agent topologies and memory regimes. We find that, although system design improves CI compliance, results are bottlenecked by incorrect LLM judgment calls: even state-of-the-art models fail to reliably filter inappropriate content or restrict transmission to authorized users. Our findings underscore the need for privacy-preserving strategies, beyond those studied in this work.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi user systems, Multi agent systems, Contextual integrity, Privacy, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tYiHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark PiSAs for measuring privacy leakage when multiple users share one agentic system. Each information item is labeled with two attributes: appropriateness (should it be used for this task?) and visibility (which users may legitimately know it?). The paper measures leakage in different channels of a multi-user system, such as gathered-information summary, inter-agent messages, and memory. Results across 3 topologies (Single, Centralized, Decentralized) and 4 memory regimes (None, Private, Shared, Hybrid) show that partitioning information into multiple agents reduces both privacy violation and completeness, while shared memory increases completeness at the cost of cross-user exposure. Overall, no studied configuration achieves acceptable privacy requirements, highlighting the need for a new approach for information sharing in multi-user agent system.

Critical strengths
  1. The paper is well-written and easy to follow. The two-label design of each attribute is very natural and clean in multi-user system. Keeping the two dimensions separate lets the benchmark measures inappropriate information flows among multiple parties, which reveals potential privacy risks that can't be captured in output-only measurement.

  2. The evaluation covers a broad range of agent topologies, memory configurations, privacy defenses, and model backbones,which is very comprehensive and produces informative results reveals interesting privacy-utility trade-offs.

Critical weaknesses
  1. Inconsistency between the benchmark's normative labels and human judgment complicate the interpretation of violation rates. Human validation suggests that the benchmark is more conservative than human annotators, potentially overestimating appropriateness violations. Higher leakage of distractors and attached private details may therefore reflect failures to follow CI, disagreement with the benchmark’s normative assumptions, or both. Testing whether this pattern persists under human-majority relabeling would strengthen the attribution to incorrect privacy judgments.

  2. The simplified simulation limits external validity. Users are represented by static information, and each scenario involves one executor pursuing a single task, with no memory accumulation across tasks. It remains unclear whether the findings generalize to natural, ongoing organizational interactions.

Questions for authors

None

4 comments

We thank the reviewer for recognizing the paper’s clear presentation, natural two-label design that captures privacy risks beyond output-only measurements, and comprehensive evaluation revealing important privacy–utility trade-offs. We address the concerns below.

Q1: Validity of appropriateness labels

We analyzed the 80 disagreements in Appendix D of the revised manuscript. They fall into two categories:

  1. Private details attached to appropriate facts (28 cases): For example, “Taylor has an HR-mediated conflict with Sasha” contains useful information for ticket allocation but also reveals a private HR-related reason. Human annotators, evaluating this attribute in isolation, consider it appropriate because it helps determine ticket assignments. However, the task only requires knowing that Taylor and Sasha cannot work together. PiSAs labels the clean constraint appropriate and the version with private details inappropriate.

  2. Speculative evidence (52 cases): Human annotators may consider speculative evidence appropriate because it appears relevant to the task. PiSAs deliberately takes a normative approach, grounded in Contextual Integrity (CI) and carefully researched, task-specific decision rubrics reflecting organizational procedures (Appendix A). Only information required under these rubrics is labeled appropriate; additional information that appears relevant but does not satisfy the decision criteria is labeled inappropriate. For example, in Severity Classification, one attribute states, “This rule has been generating alerts that consistently fail to correlate with actual incidents.” This may suggest a benign alert, but the task rules require a confirmed finding about the current incident and is therefore labeled inappropriate. These labeling choices aim to reduce subjective human biases and speculative reasoning when evaluating whether agents preserve CI, and thus these observed disagreements do not indicate incorrect labels.

Following the reviewer's concern, we also added a sensitivity analysis in Appendix D, Table 7 of the revised manuscript. On the same model outputs and human-annotated attributes, human labels reduce average V_appr from 74.8% to 62.9% (Single), 52.4% to 44.9% (Centralized), and 31.4% to 27.4% (Decentralized). Thus, while absolute violation rates change, substantial violations remain and the ordering across architectures is unchanged.

Q2: Generalization to Long-running interactions

We thank the reviewer for highlighting this limitation. PiSAs provide a controlled first step towards evaluating CI in multi-users systems. Our results show that even in this simplified single-task setting, current models struggle with CI, while shared memory increases cross-user exposure. Long running interactions introduce further risks as information accumulates and its appropriateness changes across tasks. CIMemories[1] also reports increasing privacy violations across tasks (0.1% to 9.6% over 40 tasks). We see extending PiSAs to persistent memory and evolving privacy norms as important future work, with our current benchmark providing a baseline and groundwork for these more complex settings.

[1] Niloofar Mireshghallah, Neal Mangaokar, Narine Kokhlikyan, et al. “CIMemories: A Compositional Benchmark for Contextual Integrity in LLMs.” International Conference on Learning Representations (ICLR), 2026.

AuthorsOpenReview ↗

Thank you for your feedback

We thank the reviewer for their thoughtful feedback. We have addressed the concerns raised and provided additional analyses and clarifications in our responses and revised manuscript. We would appreciate it if you could take another look and let us know if any concerns remain. If our responses have addressed your reservations, we would be grateful if you could consider updating your scores accordingly.

Dear Authors,

Thank you for the detailed reply and the new analysis. I would like to keep my original positive score, and strongly encourage you to highlight these points in the revision.

We sincerely thank the reviewer for their encouraging feedback and for taking the time to consider our response and additional analyses. We have incorporated the points discussed into the revised manuscript and will make sure they are clearly highlighted.

Reviewer_2BNSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary
  • The paper introduces a benchmark with scenarios for evaluating contextual integrity in agentic systems. It evaluates if agents untintetionally leak information while performing the tasks in these scenarios.
  • The scenarios are annotated along two dimensions: 1. Whether a piece of information should be used for the task in-hand 2. Whether a user/agent should be allowed to access it. The paper claims that these annotations enable evaluation of whether the information flow is appropriate and whether it is being shared with someone it should not be shared with.
  • The paper evaluates privacy leakage across single agent systems and multi-agent systems (centralized and decentralized) and different memory configurations, along with task success and utility across different LLM families and methods of privacy preservation including prompting to a specialized LLM trained to preserve privacy and methods that apply output filters.
  • They find that architectural isolation improves privacy preservation at the cost of task success whereas shared improves information availibility for better task success but leads to leakage to unintended users.
Critical strengths
  • I believe the two types of annotations that the benchmark provides is a valuable contribution for evaluating privacy leakage along two dimesions 1. unintended users and 2. unintended information .
Critical weaknesses
  • The benchmark has limited novelty. [1], [2], [3] show privacy leakage arising from information sharing across agents. The paper just uses evaluates different memory and agent architectures that are not part of the benchmark but are additional design choices, and the paper could do a better job of explaining what additional insight this benchmark could provide beyond that provided by prior work.
  • The findings from the paper are intuitively expected. A decentralized architecture naturally shares less information per agent which reduces the information flow and better preserves privacy at the cost of task completion. So it is not clear if the privacy leakage is better because no information is shared at all or is it better because of better adherence to CI norms.

[1] Juneja, Gurusha, et al. "Magpie: A dataset for multi-agent contextual privacy evaluation." arXiv preprint arXiv:2506.20737 (2025).

[2] Patil, Vaidehi, Elias Stengel-Eskin, and Mohit Bansal. "The sum leaks more than its parts: Compositional privacy risks and mitigations in multi-agent collaboration." arXiv preprint arXiv:2509.14284 (2025).

[3] Wang, Prince Zizhuang, and Shuli Jiang. "AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks." arXiv preprint arXiv:2604.01487 (2026).

Questions for authors
  1. Could the authors distinguish whether the benchmark provides new insights beyond those provided by evaluating memory and agent architectures on existing benchmarks?

  2. Is the reduction of privacy leakage in a decentralized multi-agent system due to better understanding of CI or just due to reduced information? If so, how does the benchmark help disentangle and quantify that?

4 comments

We thank the reviewer for recognizing the two-dimensional privacy annotation framework as a valuable contribution. We address the concerns below.

W1: Novelty

Appendix L compares PiSAs in detail with the closest multi-user benchmarks: MAGPIE [1], AgentSocialBench [3], MuPPET and PAC-Bench. We have made the reference to this appendix explicit and rewrote the Related Work section (in blue) to summarize the key differences in the Related Work section. Like PiSAs, [1] and [3] study multi-user, multi-agent settings, but PiSAs differs in four ways:

  1. Labels attached to the information. In [1] and [3], privacy rules define which information each user must not reveal in the conversation. In a shared system, information can keep propagating after it is shared, so PiSAs labels each attribute with which users may access it (visibility) and, separately, whether it should be used for the task (appropriateness).

  2. System-agnostic. These labels do not depend on any architecture, so PiSAs can compare any system design, whereas existing benchmarks provide scenarios for specific topologies.

  3. Privacy separated from utility. In [1], private information is essential to the task by design, and [3] does not guarantee that tasks can be solved without disclosure, so a violation may be the price of solving the task. Every PiSAs scenario has a valid solution that uses no inappropriate information.

  4. Normative labels. Each task family is defined by decision rules grounded in regulations or institutional policies, which determine which attributes are appropriate and which roles may hold them. This reduces the subjectivity of per-item judgments used in [1] and [3].

Patil et al. [2] studies a different problem: an adversary combines harmless-looking answers from siloed agents to infer a sensitive attribute. Leakage is adversarial and not grounded in contextual integrity, whereas PiSAs studies unintentional spillage among cooperative users of a shared system.

W1/Q1: New insights

We agree that the architectures are design choices rather than part of the benchmark. The contribution is that PiSAs’s labels make them comparable on the same scenarios, which yields these new findings:

  1. Task relevance drives violations. Unrelated personal attributes are almost never leaked (0.2%), while distractors (42.9%) and private details attached to needed facts (35.7%) often are (Section 5.3). Observing this requires labeling each attribute by its relevance to the task, on scenarios that can be solved without leaking anything, so that leakage reflects the model’s judgment rather than the task’s demands.

  2. Data partitioning does not solve the problem. Multi-agent topologies reduce violations mostly by sharing less, so completeness drops, and the failure rate stays at 79.6–100% (Table 2).

  3. Memory trades appropriateness for visibility violations. In the Decentralized system, memory lowers appropriateness violations (9.9% to 6.4%) but raises visibility violations from 14.9–28.0% to 35.6–50.0% (Table 11), which would look like a pure improvement without visibility labels.

  4. PiSAs does not saturate where [3] does. On [3], the only benchmark we could adapt to our evaluation (Appendix L), appropriateness violations drop to almost zero with a privacy prompt, while PiSAs still shows 11% on the same system and topology.

Q2: Decentralized Systems

We agree that decentralization limits the information initially available to each agent, making it a natural mitigation strategy to study. However, partitioning information does not make it unavailable, as agents can communicate to gather information needed to complete a task. Ideally, agents should exchange only appropriate information with authorized recipients, achieving both high task completeness and low privacy violations. Our results show that current models struggle to make this distinction. While decentralization reduces V_vis, the reduction in V_appr comes at the cost of lower task completeness, while substantial appropriateness violations still remain. This suggests that models struggle to selectively share the right information, instead often sharing too little or disclosing inappropriate information. Measuring V_vis, V_appr, and completeness together helps PiSAs distinguish the benefits of information partitioning from the model's ability to make CI-aware decisions. Hence we agree that Decentralized topology does not make the system understand better CI norms. This understanding is tied with the choice of LLM backbone, that decides what can be shared during agent communication. Table 2 shows that under the same decentralized system, GPT-Astra achieves the highest completeness (69.4%) with low V_appr (11.9%) compared with 60.4% completeness and 47.5% V_appr for Kimi-K2.5, indicating that this backbone has a better understanding of CI norms.

AuthorsOpenReview ↗

Thank you for your feedback

We thank the reviewer for their thoughtful feedback. We have addressed the concerns raised and provided additional analyses and clarifications in our responses and revised manuscript. We would appreciate it if you could take another look and let us know if any concerns remain. If our responses have addressed your reservations, we would be grateful if you could consider updating your scores accordingly.

Thank you to the authors for the detailed response and revisions.

W1: Novelty. I appreciate the expanded comparison with prior benchmarks, particularly the clarification regarding dual appropriateness and visibility annotations. However, I remain somewhat unconvinced by the significance of the novelty claims. While shared multi-user infrastructure represents a different setting, many practical multi-agent systems operate with isolated agents and private memory, and it remains unclear what new privacy challenges this setting introduces beyond existing multi-agent information-flow problems. Additionally, evaluating multiple agent architectures is not inherently novel, and the authors have not sufficiently established why existing benchmarks could not be adapted for similar comparisons. Finally, guaranteeing privacy-preserving solutions is a useful insight, but its novelty relative to prior work remains unclear. I view the dual annotations as the strongest contribution, although the extent to which they enable new insights remains uncertain.

Q2: Decentralized systems. I appreciate the clarification that decentralization primarily reduces information exposure rather than improving CI reasoning. This addresses my concern about the interpretation of the architectural comparisons. However, the finding that partitioning reduces privacy violations at the cost of task completeness is largely expected. A more detailed analysis of when agents fail to retrieve necessary information versus when they inappropriately disclose information would provide stronger insights into the limitations of current models. I do not expect additional experiments during the rebuttal period, but believe this would be a valuable direction for future revisions.

Overall, the rebuttal clarifies the contributions and addresses some of my concerns, but my reservations about the significance of the benchmark's novelty and empirical findings remain. So I will maintain my score.

AuthorsOpenReview ↗

We thank the reviewer for their additional feedback

W1: Novelty

We thank the reviewer for acknowledging the dual annotations as a novel contribution. Below, we highlight PiSAs’ novel contributions by comparing them with the four closest multi-user benchmarks (MAGPIE, AgentSocialBench, MuPPET, PAC-Bench; Appendix L) and present the new insights enabled by each.

Novelty 1: Independent visibility and appropriateness labels for every attribute.

Prior work: MAGPIE identifies each user’s private information, PAC-Bench defines privacy rules for each agent, and AgentSocialBench lists facts agents must not share. MuPPET specifies which participants should not learn certain facts, but evaluates a single agent reply, with no information flowing between agents.

New Insights: (a) Neither dimension alone captures all CI violations: visibility misses inappropriate information shared among authorized users, while appropriateness misses required information exposed to unauthorized users. (b) Partitioning improves one dimension while worsening another: compared with Centralized, Decentralized reduces the former (5.5% → 2.5%) but increases the latter (10.7% → 11.5%) (Section M.5).

Novelty 2: Shared infrastructure, not only isolated agents.

Prior work: All four study isolated agents, and none includes shared memory and agents. Why it matters: when users share a system, privacy is not only about what an agent reveals, but about which users each fact can reach. The shared setting is relevant in workplace scenarios, where useful information is spread across users and needs to be gathered to solve tasks. In this context, optimal system design is an open research question as it is not straightforward which design would give the best privacy and utility trade-off.

New Insights: Shared memory shifts risk instead of removing it: in the Decentralized system, it lowers appropriateness violations in agent messages (9.9% → 6.4%) but raises visibility violations (14.9–28.0% → 35.6–50.0%, Table 12).

Novelty 3: The same scenarios and labels support different system designs.

Prior work: Each benchmark is built around a fixed interaction setting (MAGPIE on negotiations, PAC-Bench on two privately owned agents, MuPPET on a single assistant in a group chat, and AgentSocialBench on settings such as hub-and-spoke coordination), whereas PiSAs defines tasks and labels independently of topology and memory, so the same scenarios run on any system design.

New Insights: (a) privacy mitigations do not compound: Privasis and PrivacyChecker substantially reduce appropriateness violations in the single-agent system, but bring little benefit in multi-agent ones (Section 5.4); (b) systems lose required information at different points: Centralized mainly because it is never obtained (35.9%), Decentralized because the executor’s agent omits facts it already has (40.3%) (Section M.6).

Novelty 4: Policy-grounded tasks with a guaranteed privacy-preserving solution.

Prior work: MAGPIE makes private information essential to the task; the other three do not guarantee by construction that tasks can be solved without compromising privacy.

New Insights: (a) every failure reflects the system rather than the task, and can be attributed to missing required information, inappropriate use, or unauthorized disclosure; (b) violations are driven by task relevance: unrelated personal attributes are almost never leaked (0.2%), but distractors that look relevant often are (42.9%) (Section 5.3).

 

Q2: Decentralized Systems

Our findings go beyond the expected privacy–completeness trade-off: Our new analyses show fewer appropriateness-only but more visibility-only violations in the decentralized system (Sec. M.5) and, following the reviewer’s suggestion, we also trace where appropriate and inappropriate information is lost or disclosed (Sec. M.6):

  • Appropriate information fails to reach the writer more often in the centralized system (35.9%) compared with 14.7% in the decentralized system.
  • Agents also fail to use information they already have. 18.0% of the appropriate attributes in the centralized system reach the coordinator and 40.3% reach the executor in the decentralized system, but are omitted from their output.
  • Limiting information access does not prevent inappropriate disclosure. Writers include about 41–48% of the inappropriate information reaching them in their outputs.

Furthermore, Fig. 3b shows that agents rarely disclose unrelated personal information (0.2%) but frequently disclose seemingly relevant distractors (42.9%) and private details attached to useful information (35.7%). This suggests that models struggle to distinguish apparent task relevance from appropriate use. This is a new result not studied in previous works.

Together, these findings identify distinct failures in retrieving, selecting, and sharing information that cannot be explained by data and agent partitioning alone.

Reviewer_sz8xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes the first multi-user privacy benchmark, PiSAs, based on the "Contextual Integrity " theory. The paper reveals that current mainstream large-scale models and architectures commonly suffer from severe cross-user data contamination when dealing with task-related but confidential details. Furthermore, existing physical isolation and defensive cue words come at the cost of significantly sacrificing task completion, failing to completely resolve the privacy-utility trade-off dilemma.

Critical strengths
  1. The paper extends traditional single-user privacy issues to multi-user shared agent infrastructures, focusing on the improper propagation of information among different user agents, internal communications, and shared memories.
  2. The paper's experiments cover a wide range of system configurations.The paper compares eight LLMs, three agent topologies, three memory configurations, different privacy cue strengths, and specific privacy mitigation schemes.For example, Table 2 on page 6 shows significant differences between different topologies in terms of privacy and task completion capabilities; pages 8–9 further analyze the impact of memory configuration, privacy cue strengths, and different information categories.
  3. The paper specifically validates annotation and automated evaluation. For example, Appendix D uses manual annotation to validate privacy labels, and Appendix E compares and analyzes the effectiveness of LLM-as-a-Judge.
Critical weaknesses
  1. The core contribution of this paper is the separate definition of task appropriateness and user visibility. Therefore, whether these two types of labels accurately reflect privacy norms in real-world scenarios directly determines the credibility of the benchmark. It is particularly noteworthy that in 81 cases where there was disagreement regarding the appropriateness label, 80 cases involved individuals who believed the information was suitable for the task, while the benchmark marked it as inappropriate. This indicates a significant discrepancy.
  2. The paper sets the Visibility Violation to 100% in a single-agent architecture, arguing that all users share the same Agent, and the Agent has access to all users' information. However, if a shared Agent stores data from multiple users but strictly adheres to access permissions when answering questions, it still carries potential access control risks without necessarily resulting in actual cross-user information disclosure. This leads to a difference in visibility metrics between single-agent and multi-agent systems that is predetermined by the system design.
  3. The paper uses shared memory and hybrid memory as key experimental subjects and finds that using memory improves information integrity but also leads to more cross-user information contamination. However, Appendix G explicitly states that "Memory is scoped to a single run — nothing carries over across scenarios." This means that this type of cross-task memory contamination may accumulate with increasing runtime and number of tasks.
Questions for authors
  1. The paper's human validation results show a systematic discrepancy between benchmark labels and human judgment, particularly in the appropriateness label category. Of the 81 discrepancies, 80 involved situations where humans deemed the information appropriate, but the benchmark deemed it inappropriate. Could this conservative labeling strategy systematically overestimate the privacy violation rate? Could the authors reassess the main experimental findings using manually corrected labels and extend human validation to all six domains?
  2. The visibility violation rate for single-agent architectures is fixed at 100% based on system design, while for multi-agent systems it is evaluated through actual communication and memory exposure. Does this definition confuse potential information accessibility with actual information leakage, thus affecting the fairness of comparisons between different agent architectures? Can the authors report potential exposure risk and actual unauthorized information disclosure rate separately under consistent user access assumptions?
  3. The paper argues that shared memory is a significant source of cross-user information leakage. However, the memories in the experiments only exist during the execution of a single task and are not persistently retained across tasks or scenarios. Does this mean these results still hold true in multi-user, long-running agent systems with persistent memory? Could the authors add continuous task experiments and introduce appropriateness and visibility constraints that vary with the task to assess the risk of cross-task memory contamination?
4 comments

We thank the reviewer for recognizing the paper’s focus on cross-user information propagation, comprehensive evaluation across models and system configurations, and validation of privacy annotations and LLM-as-a-Judge evaluation. We address the concerns below.

Q1: Validity of appropriateness labels

We analyzed the 80 disagreements in Appendix D of the revised manuscript. They fall into two categories:

  1. Private details attached to appropriate facts (28 cases): For example, “Taylor has an HR-mediated conflict with Sasha” contains useful information for ticket allocation but also reveals a private HR-related reason. Human annotators, evaluating this attribute in isolation, consider it appropriate because it helps determine ticket assignments. However, the task only requires knowing that Taylor and Sasha cannot work together. PiSAs labels the clean constraint appropriate and the version with private details inappropriate.

  2. Speculative evidence (52 cases): Human annotators may consider speculative evidence appropriate because it appears relevant to the task. PiSAs deliberately takes a normative approach, grounded in Contextual Integrity (CI) and carefully researched, task-specific decision rubrics reflecting organizational procedures (Appendix A). Only information required under these rubrics is labeled appropriate; additional information that appears relevant but does not satisfy the decision criteria is labeled inappropriate. For example, in Severity Classification, one attribute states, “This rule has been generating alerts that consistently fail to correlate with actual incidents.” This may suggest a benign alert, but the task rules require a confirmed finding about the current incident and is therefore labeled inappropriate. These labeling choices aim to reduce subjective human biases and speculative reasoning when evaluating whether agents preserve CI, and thus these observed disagreements do not indicate incorrect labels.

Following the reviewer's concern, we also added a sensitivity analysis in Appendix D, Table 7 of the revised manuscript. On the same model outputs and human-annotated attributes, human labels reduce average V_appr from 74.8% to 62.9% (Single), 52.4% to 44.9% (Centralized), and 31.4% to 27.4% (Decentralized). Thus, while absolute violation rates change, substantial violations remain and the ordering across architectures is unchanged.

Q1: Human verification on all domains

We used SuperAnnotate (https://www.superannotate.com/) to recruit domain experts for our human validation study. The strong agreement between expert annotations and model-based labels on the evaluated sample supports the reliability of our scenario generation and labeling process. Extending validation to all domains would require months of contracting, training, and quality control, at substantial cost, beyond the rebuttal period.

Q2: Potential vs. Actual Disclosure in Single

We thank the reviewer for raising this important distinction. The 100% V_vis for Single measures potential exposure by design, since the shared agent has access to all information from every user. Although our experiments are single-task and non-adversarial, such unrestricted access poses significant risk in real-world systems involving long-running, multi-tasks, or adversarial settings. Following the reviewer's suggestion, we now separately report observed output disclosure in Appendix M.3 (Table 16) and Section 5.5 of the revised manuscript. Assuming the final output is visible to all users, we measure the fraction of visibility-restricted inappropriate attributes disclosed in it as V^out_vis. Averaged across eight backbones, the observed disclosure rates in the final output are 14.7% for Single, 12.5% for Centralized, and 8.2% for Decentralized, providing a consistent comparison of actual output disclosure across architectures.

Q3: Persistent Memory and Long-Running Tasks

PiSAs focuses on single-task interactions, where we already observe substantial CI violations and increased exposure through shared memory. Together with [1], we expect violation rates to rise across tasks. We conducted a preliminary two-task experiment with 10 JIRA Allocation → Meeting Scheduling pairs, where information appropriate in Task 1 became inappropriate in Task 2. Under Decentralized with High privacy instructions, old-information resurfacing increased from 1.5% with shared memory reset between tasks to 3.8% otherwise. However, these tasks were largely unrelated, leaving little opportunity for old information to become relevant again. Given our finding that task relevance strongly drives privacy violations (Fig 3b), a more meaningful evaluation requires carefully designed sequences of related tasks with evolving privacy norms. We consider this important future work.

[1] Niloofar Mireshghallah, et al. “CIMemories: A Compositional Benchmark for Contextual Integrity in LLMs.” ICLR, 2026.

AuthorsOpenReview ↗

Thank you for your feedback

We thank the reviewer for their thoughtful feedback. We have addressed the concerns raised and provided additional analyses and clarifications in our responses and revised manuscript. We would appreciate it if you could take another look and let us know if any concerns remain. If our responses have addressed your reservations, we would be grateful if you could consider updating your scores accordingly.

I thank the authors for their detailed response, which addresses most of my concerns. I have therefore raised my soundness score from 2 to 3 and maintain my overall positive assessment.

We sincerely thank the reviewer for their thoughtful feedback, for taking the time to carefully consider our response, and for their positive reassessment of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We thank all reviewers for their constructive feedback.

We appreciate that the reviewers highlighted:

  1. The extension of privacy evaluation to shared multi-user agentic systems, capturing information propagation through internal communication and shared memory (sz8x).
  2. The valuable two-dimensional privacy annotation framework, separately capturing unintended recipients and inappropriate information flows, including risks missed by output-only evaluation (2BNS, tYiH).
  3. The comprehensive evaluation across models, agent topologies, memory configurations, and privacy defenses, revealing important privacy–utility trade-offs (sz8x, tYiH).
  4. The validation of privacy annotations and LLM-as-a-Judge evaluation (sz8x).
  5. The clear, well-written presentation (tYiH).

We have revised the manuscript to address the reviewers' concerns, clarified our contributions relative to prior work, and conducted the following additional analyses (changes are highlighted in blue in the manuscript):

RevisionKey findingPaper locationReviewer concern
Detailed analysis of human-label disagreements80 disagreements: 28 involving private details attached to appropriate facts and 52 involving speculative evidence, all labels unchanged under normative design of PiSAsAppendix Dsz8x (Q1); tYiH (W1)
Re-evaluation with human appropriateness labelsAppropriateness violations remain high, ordering across architectures unchangedAppendix D, Table 7sz8x (Q1); tYiH (W1)
Visibility violations measured in task outputVisibility violation rates: 14.7% Single, 12.5% Centralized, 8.2% DecentralizedSection 5.5, Appendix M.3, Table 16sz8x (Q2)
Clarify benchmark novelty wrt prior workIndependent visibility and appropriateness labels; privacy in shared infrastructure; system-agnostic scenarios; policy-grounded tasks with guaranteed privacy-preserving solutionsContributions, Related Work, Appendix L2BNS (Q1, Q2)
Effects of information partitioningCompared with Centralized, Decentralized has fewer appropriateness-only but more visibility-only violations; appropriate facts more often reach the writer, but are more often omitted despite being available.Appendix M.5–M.62BNS (Q1, Q2)

We respond to each reviewer's specific concerns in detail below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.