acceptodds
Under review as a conference paper at ICLR 2027

Contextual Privacy in Agentic Workflows is a Systems Problem: A Benchmark, Auditor & Guard

Abstract

Agentic systems handle sensitive personal data on users’ behalf, often across multiple tools and stages. Privacy work has focused on what the user shares and what the agent finally outputs. The trajectory between the user’s input and the agent’s final output remains largely unmeasured, even though it is where much of the exposure occurs: agents request more data than the task requires, tools return it indiscriminately, and private information reaches parties and stages where it does not belong. We introduce Σ-Bench, a benchmark of 677 privacy-sensitive scenarios grounded in everyday personal life, paired with PrivacyTrace, a typed-flow graph that records every data flow violation across four execution stages. Across 16 frontier agent models on a stratified 85-scenario subset, we find pipeline viola- tions in 83–99% of executions, compared with output leakage of 33–55%. Thus, output-only evaluation undercounts exposure by at least 1.5×, over 2× for most models. While privacy-aware prompting does not reliably reduce privacy violation, higher task success is also associated with greater leakage. We further introduce PrivacyTraceGuard, our runtime guard, that allows, filters, or blocks each flow at its boundary. It intervenes at every pipeline stage and reduces both pipeline and output violations. Filtering reduces pooled output leakage from 43.8% to 17.1%, and blocking to 6.7%, while pipeline violations remain 80.1% and 77.5%, respectively. These results show that privacy is a property of the full execution and requires boundary-level auditing and intervention. Code, data, and trace artifacts will be released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent privacy, privacy leakage, benchmark, contextual integrity, trustworthy AI, tool-use agents, AI safety, runtime guardrails, LLM-as-a-judge, information flow, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_syyPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that privacy in agentic workflows should be evaluated over the full execution pipeline rather than only the final output. The authors introduce Σ-Bench, a benchmark of privacy-sensitive personal-data scenarios, and PrivacyTrace, a graph representation that records information flows across instruction, query, tool-response, and output boundaries. They evaluate 16 agent models and find that pipeline violations are much more frequent than final-output leaks.

Critical strengths
  1. The paper addresses an important limitation of output-only privacy evaluation. In agentic systems, sensitive data may be exposed through tool calls, tool responses, intermediate context, or logs even when the final response looks safe.
  2. The empirical study is broad, covering many frontier and open-weight models, multiple privacy domains, and several intervention settings.
Critical weaknesses
  1. The paper uses a conservative rule that a transfer is appropriate only if it is both needed for the task and fitting for the recipient. This makes sense as a strict auditing stance, but the paper often presents every upstream exposure as a privacy violation without sufficiently distinguishing trusted local processing, tool-side over-return, and actual disclosure to an external recipient.

    For example, in the opening sick-leave scenario, saying “out sick” is itself health-related information requested by the user to be sent to the manager, while fertility and therapy appointments are clearly more sensitive and not task-relevant.

    The paper would be stronger if it clarified this distinction rather than treating the example mainly as a binary private/not-private contrast.

  2. The benchmark relies heavily on synthetic personas, synthetic scenarios, and LLM-assisted scenario generation and judging. The authors report validation steps and cross-judge checks, but there is no full human annotation study. Since contextual privacy norms are subtle and scenario-dependent, this weakens confidence in the ground truth labels and in claims about what is “appropriate” across domains.

  3. PrivacyTraceGuard has limited practical effectiveness under the paper’s own metric. Even with filtering or blocking, pipeline violation rates remain around 80% and 77.5%, respectively. This raises a concern that the proposed intervention mostly shifts where leakage is observed rather than providing a robust solution to contextual privacy in agentic workflows.

Questions for authors

Please refer to the Weaknesses.

3 comments

We thank the reviewer for the review, and for recognizing the breadth of the study across models, privacy domains, and interventions. Point-by-point response below.

Concern 1: Upstream exposure is not separated from disclosure to an external recipient (W1a, W1b).

We agree these are different, and the four boundaries keep them apart. External disclosure is the output boundary, which LR measures, and tool over-return is the response boundary. Local processing is whatever lands inside the agent, namely the user's instruction and the tool's response, whereas the query lands in the tool and the output lands with the recipient. Because each boundary is scored on its own, every trust stance a deployer might take maps to a metric we report, and the table below computes all four from the same runs (new runs on the expanded benchmark described in Concern 3, mean of eight models).

What the deployer trustsBoundaries that countRate
Nothing (our default)instruction, query, response, output96.4 (PVR)
The toolsinstruction, response, output95.0
The agentquery, output79.0 (PVR_QO)
The agent and the toolsoutput48.9 (LR)

Trusting the tools alone changes little, because what they return still enters an untrusted agent, while trusting the agent lowers the rate the most. Even with both trusted, about half of runs send an unneeded item to an external recipient, so the main finding does not rest on the zero-trust default.

This flexibility is part of the design, because a deployer chooses which boundaries count for their setting, and the guard can be set separately at each one to allow, filter, or block. A deployer who hosts the agent themselves, for example, can treat what reaches the agent as safe and check only what the agent sends out, its queries to tools and its messages, which is exactly PVR_QO. For the camera-ready, Section 2 will present this table, with the zero-trust default as one of the four rows, and will explain how a deployer sets the boundaries to match their trust zone.

Concern 2: The sick-leave example (W1c, W1d).

The reviewer is right that being out sick is health information, so our sentence saying the instruction "contains no sensitive information" was wrong. The instruction is still appropriate, because the task needs it and the manager is the right person to receive it, which is not true of the fertility and therapy appointments. We will rewrite the sentence and relabel the instruction edge in Figure 1 as appropriate because it is needed and fits the recipient, so that the example applies our rule instead of appearing to ignore it.

Concern 3: The labels and judge rely on LLMs, without a full human study (W2).

Σ-Bench is synthetic, as are PrivacyLens [1] and CIMemories [2], because real user data with privacy labels would itself be risky to collect and share. Since the labels are synthetic too, we check them in three ways. First, every scenario passes structural rule checks, LLM critique, and an automated check that its task, app data, and contact book agree, and all 750 scenarios of the expanded benchmark described below pass; the authors also went through the benchmark persona by persona, reviewing every scenario the checks flagged and holding back any that did not yet pass. Second, a rule-based evaluator that flags only exact ground-truth items, and so cannot over-flag, still finds PVR more than twice LR (68.0 against 30.7 on the new runs, and 65.2 against 26.0 in Appendix I.6). Third, a proprietary judge and an open-weights one, gpt-5.5 and gpt-oss-120b, agree with Cohen's κ between 0.63 and 0.88 across the four boundaries, after raising the open-weights judge's output-length limit, which in the submission was set too low for a reasoning model and cut its answers short.

Since submission, we have expanded the benchmark from 677 to 750 scenarios and the evaluation subset from 85 to 100, and every scenario now passes an automated check that its task, app data, and contact book agree. We also gave the tools the narrower filters that real APIs provide, such as selecting calendar events by date or tag, which removes the 11.1% floor that Appendix B reports for a perfect agent under the original tools, and added to every scenario an oracle trajectory that completes the task with PVR_QO = 0. With these filters no tool is forced to return an item the task does not need, so the oracle's remaining PVR of about 20% comes entirely from instructions in which the user already shares a sensitive detail the task does not need, which neither the agent nor the tools can undo.

We ran the experiments on the expanded benchmark, using the 100-scenario subset with three trials per scenario and eight of the sixteen models so far; the other eight will follow in the revision.

ModelLRPVR_QOPVRTSR
Claude Haiku 4.547.073.088.779.7
Claude Opus 4.748.385.799.792.0
GPT-5.552.381.099.397.0
Granite 4.1 30B39.469.294.873.0
GLM-4.6V51.079.897.783.3
MiniMax M2.559.784.399.792.0
Mistral Large 345.082.399.385.3
Kimi K2.748.376.392.077.0
Mean48.979.096.484.9

Each number is the percentage of runs, over 100 scenarios with three trials each, and the standard error is 3 to 5 points per model, as in Table 1. Absolute levels differ from Table 1 because the scenario set has changed, but the paper's main findings hold on the new runs. PVR is 1.7 to 2.4 times LR for every model, and PVR_QO, which counts only the boundaries the agent controls, is still about 1.6 times LR and still separates the models. The response boundary is still where violations are most frequent, and on the four models with prompting results so far, a privacy-aware prompt does not lower LR. Kimi K2.7 replaces K2.5, which is no longer available to us.

We also ran a human study for this rebuttal. Two annotators independently labeled the same 50 pipeline steps, sampled from the submitted runs of all sixteen models and covering all four boundaries (11 to 16 steps each), without seeing the judge's verdicts. They agree with each other with Cohen's κ of 0.72, so the labels people give are consistent. With both applying the same written rule at each boundary, they agree with the judge with an average κ of 0.73 at the instruction boundary, 0.42 at response, and 0.82 at output. At the query boundary agreement is lower (κ 0.23), and in all 11 disagreements there, the annotators flagged a query that the judge allowed. The query rates we report therefore understate, if anything, how often agents ask for more than they need. The revision will extend the study to more steps per boundary.

Refs.

[1] Shao, Y., Li, T., Shi, W., Liu, Y., & Yang, D. (2024). Privacylens: Evaluating privacy norm awareness of language models in action. Advances in Neural Information Processing Systems, 37, 89373-89407.

[2] Mireshghallah, N., Mangaokar, N., Kokhlikyan, N., Zharmagambetov, A., Zaheer, M., Mahloujifar, S., & Chaudhuri, K. (2026, April). CIMemories: A compositional benchmark for contextual integrity in LLMs. In International Conference on Learning Representations (Vol. 2026, pp. 95513-95532).

Concern 4: The guard only shifts where leakage appears (W3).

If the guard only shifted leakage, the total would stay the same, but it drops instead, at every boundary. The table below gives the new runs for the eight models, on the expanded benchmark described in Concern 3.

Mean of eight modelsInstructionQueryResponseOutput (LR)PVR_QOPVRTSR
No guard19.460.890.348.979.096.484.9
Guard, filter2.140.956.812.247.275.966.2
Guard, block1.039.220.01.840.048.528.2

Violations fall at every boundary, for every model, so what the guard removes at one boundary does not reappear at another. Response violations fall too, because the new runs count only what reaches the agent, and the guard stops a withheld return from reaching it. The cost is task success, as in the submitted Table 2. Most of the drop comes from what the guard withholds. When it holds back a request or a tool's return, the agent often cannot finish, and under block a flagged instruction is refused outright, which ends about one run in five. Filter therefore keeps two thirds of tasks working while cutting output leaks by about three quarters, and block brings PVR below 50 at a larger cost, which is why we recommend filter as the default and block for settings where a leak matters more than finishing the task. The guard also turns leaky completions into clean ones. On the 24 of the 100 scenarios where every unguarded run that completes the task also leaks in its output, a guarded run completes the task without the leak on 22. For the camera-ready, we will report these results for all sixteen models and lead the guard results with LR and PVR_QO.

We hope these responses address the reviewer's concerns and are happy to discuss any point further during the discussion period.

Reviewer_MyVnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing privacy evaluations for LLM agents focus on endpoint risks (e.g. what the agent finally shares / outputs) and ignores the potentially inappropriate information flow during execution. This paper instead adopts a zero-trust perspective, assuming no participant or boundary is by default safe. It introduces a synthetic benchmark of privacy-sensitive scenarios paired with a typed-flow graph that records every data transfer across trust boundaries (instruction, query, response, output). Experiment results show that Pipeline Violation Rate (PVR) is significantly higher than the output-only Leak Rate (LR), showing that privacy violations can be greatly underestimated in with output-only measurement. For defense, privacy-aware prompting fails to reduce privacy violation, while PrivacyTraceGuard significantly reduce LR but PVR remains high, indicating that the current approaches are not sufficient to address inappropropriate information flows in agentic workflows.

Critical strengths
  1. The framing of privacy violation as a process rather than a static outcome is very novel and well-motivated. This is especially important when the agents are deployed on third-party platforms and the tools they use are external, which is also the most common case for today's LLM agent users.

  2. The evaluation environment, built on Meta's ARE, enables realistic agent run simulation. This makes the result more convincing than runs that rely entirely on LLM-emulated tools, as in the original setup of PrivacyLens.

  3. The results are very strong and support the main claims of the paper very well. The huge distinction between PVR and LR highlights the potential privacy risks that output-only measurement fails to capture. Defense approaches also reveal an inherent privacy-utility trade-off for current LLM agents, underscoring the need for a win-win privacy protection mechanism.

Critical weaknesses
  1. The realism of these LLM generated scenarios should be further verified. Under the paper's definition, we assume that all data transfers across trust boundaries are risky; Meanwhile, it also delibrately places task-relevant and senitive but irelevant information together, creating conditions that favor over-retrievals. Therefore, the gap between PVR and LR seem to be a result of deliberate design instead of widely observed phenomena. The authors should phrase their claim more carefully, or show results on some naturally occuring tasks to strengthen this it.

  2. The evaluation applies a conservative zero-trust policy across all four boundaries, but the privacy implications of a transfer may depend on whether the agent and the tools operate within the user’s trusted environment, which are two independent deployment choices. A systematic comparison of configurations that trust the agent, the tools, or both would clarify how strongly the findings depend on the chosen trust assumptions.

  3. The reliability of scenario ground truth and violation judgments needs stronger human validation. Scenarios, ground truth labels and violation detection are currently primarily done by LLM with limited / no human verification. Agreement among different models are not so high. So it's questionable how reliable the findings are, given limited human involvement in the process.

Questions for authors

None

3 comments

We thank the reviewer for the thoughtful review. Point-by-point response below.

Concern 1: The PVR–LR gap comes from deliberate scenario design (W1).

We place needed and unneeded data in the same apps on purpose, because that is how personal data is stored, with a work meeting and a medical appointment in the same calendar and a project thread and a message from a lawyer in the same inbox. The design must not, however, force the agent to read the unneeded item, and two results show that it does not. An oracle completes every task without asking for the unneeded item, and on half of the scenarios a real agent does the same.

Since submission, we have expanded the benchmark from 677 to 750 scenarios and the evaluation subset from 85 to 100, and every scenario now passes an automated check that its task, app data, and contact book agree. We also gave the tools the narrower filters that real APIs provide, such as selecting calendar events by date or tag, which removes the 11.1% floor that Appendix B reports for a perfect agent under the original tools, and added to every scenario an oracle trajectory that completes the task with PVR_QO = 0. With these filters no tool is forced to return an item the task does not need, so the oracle's remaining PVR of about 20% comes entirely from instructions in which the user already shares a sensitive detail the task does not need, which neither the agent nor the tools can undo.

We ran the experiments on the expanded benchmark, using the 100-scenario subset with three trials per scenario and eight of the sixteen models so far; the other eight will follow in the revision.

ModelLRPVR_QOPVRTSR
Claude Haiku 4.547.073.088.779.7
Claude Opus 4.748.385.799.792.0
GPT-5.552.381.099.397.0
Granite 4.1 30B39.469.294.873.0
GLM-4.6V51.079.897.783.3
MiniMax M2.559.784.399.792.0
Mistral Large 345.082.399.385.3
Kimi K2.748.376.392.077.0
Mean48.979.096.484.9

Each number is the percentage of runs, over 100 scenarios with three trials each, and the standard error is 3 to 5 points per model, as in Table 1. Absolute levels differ from Table 1 because the scenario set has changed, but the paper's main findings hold on the new runs. PVR is 1.7 to 2.4 times LR for every model, and PVR_QO, which counts only the boundaries the agent controls, is still about 1.6 times LR and still separates the models. The response boundary is still where violations are most frequent, and on the four models with prompting results so far, a privacy-aware prompt does not lower LR. Kimi K2.7 replaces K2.5, which is no longer available to us.

These runs also show that real agents find the oracle's clean path, since on half of the scenarios at least one model completes the task with no query or output violation, and 15% of all runs do so. In most runs, however, the agent still asks for more than the task needs, even though a narrower request is available.

We agree that the size of the gap depends on how often needed and sensitive data sit together, which Σ-Bench sets by design. Measuring it in real users' data would require access to their calendars, inboxes, and messages, along with their consent, which in practice only a service provider that already holds that data could obtain. This is also why other privacy benchmarks for language models use synthetic data. The camera-ready will state the claim under this condition and report the clean-path results above.

Concern 2: The findings depend on the zero-trust assumption (W2).

We agree that trusting the agent and trusting the tools are separate deployment choices. Each choice changes which boundaries count, and since every boundary is scored on its own edge, all four configurations can be computed from the same runs, as the table below shows for the new runs (mean of eight models).

What the deployer trustsBoundaries that countRate
Nothing (our default)instruction, query, response, output96.4 (PVR)
The toolsinstruction, response, output95.0
The agentquery, output79.0 (PVR_QO)
The agent and the toolsoutput48.9 (LR)

Trusting the tools alone changes little, because what they return still enters an untrusted agent, while trusting the agent lowers the rate the most. Even with both trusted, about half of runs send an unneeded item to an external recipient, so the main finding does not rest on the zero-trust default.

This flexibility is part of the design, because a deployer chooses which boundaries count for their setting, and the guard can be set separately at each one to allow, filter, or block. A deployer who hosts the agent themselves, for example, can treat what reaches the agent as safe and check only what the agent sends out, its queries to tools and its messages, which is exactly PVR_QO. For the camera-ready, Section 2 will present this table, with the zero-trust default as one of the four rows, and will explain how a deployer sets the boundaries to match their trust zone.

Concern 3: The labels and judgments need stronger human validation (W3).

We checked the scenario labels and the judge's verdicts separately.

For the labels, every scenario passes structural rule checks and LLM critique as well as the consistency check described in Concern 1, and all 750 pass. The authors then went through the benchmark persona by persona, reviewing every scenario the checks flagged and holding back any that did not yet pass.

For the judge, a rule-based evaluator that flags only exact ground-truth items, and so involves no LLM, still finds PVR more than twice LR on the new runs (68.0 against 30.7). A proprietary judge and an open-weights one, gpt-5.5 and gpt-oss-120b, also agree with each other, with Cohen's κ of 0.72 at the instruction boundary, 0.68 at query, 0.88 at response, and 0.63 at output. The submission reported lower values (0.26 at query and 0.22 at output) because the open-weights judge's output-length limit was set too low for a reasoning model and cut its answers short; with the limit raised, agreement is as above. The much smaller Granite 4.1 30B judge agrees less at query and output (κ near 0.3), and that is why we score with a frontier judge.

We also ran a human study for this rebuttal. Two annotators independently labeled the same 50 pipeline steps, sampled from the submitted runs of all sixteen models and covering all four boundaries (11 to 16 steps each), without seeing the judge's verdicts. They agree with each other with Cohen's κ of 0.72, so the labels people give are consistent. With both applying the same written rule at each boundary, they agree with the judge with an average κ of 0.73 at the instruction boundary, 0.42 at response, and 0.82 at output. At the query boundary agreement is lower (κ 0.23), and in all 11 disagreements there, the annotators flagged a query that the judge allowed. The query rates we report therefore understate, if anything, how often agents ask for more than they need. The revision will extend the study to more steps per boundary.

For the camera-ready, Appendix I will report the human study, the corrected agreement table, and the label checks.

We hope these responses address the reviewer's concerns and are happy to discuss any point further during the discussion period.

Dear Authors,

Thank you for the detailed reply and the new experiments. I think they have resolved most of my concerns. I keep my original positive score and look forward to seeing these analysis in the camera-ready version.

Reviewer_eMtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that privacy in agentic systems should be measured across the whole execution rather than at the final output. The authors define four trust boundaries in a single agent run (user→agent instruction, agent→tool query, tool→agent response, agent→recipient output), build Σ-Bench, a synthetic benchmark of privacy-sensitive scenarios that runs live in a tool-calling simulator, and record every transfer as an edge in a typed graph called PrivacyTrace. Across 16 models they report that violations at some boundary occur in 83–99% of runs against output-only leak rates of 33–55%, that privacy-aware system prompts do not reduce leakage, and that a runtime per-edge guard lowers output leakage substantially but leaves pipeline violations near 80%.

Critical strengths
  • The core question is a good one and the Emma example is an unusually clear illustration of it: an agent can produce a clean email while the user's fertility appointment has already travelled through two hops it had no reason to touch.
  • The privacy-trap taxonomy is a reusable contribution. Seven structural co-location patterns where sensitive and routine items share a timestamp, a folder, a first name or a search keyword is a sharper construct than injecting irrelevant data, because it builds cases where no narrow query exists at all.
  • The empirical work is thorough: 16 models, 3 trials, 5 conditions, a full-benchmark replication, a rule-based lower bound, three judge models, and a local open-weights judge substitution. The limitations section is honest about most of this.
Critical weaknesses
  • Novelty is thinner than the framing claims, because the related work is mischaracterized. The paper states that across this line of work "evaluation is endpoint-based: each benchmark scores the agent's final output," but AgentDAM measures data minimization over the agent's actions end-to-end, which already covers the agent→tool query boundary. Decomposing their own per-stage numbers (instruction 19%, query 50%, response 73%, output 44%), prior work covers output, data-minimization work covers query, and the instruction stage is the user's and is pinned at 17–20% for all 16 models by scenario design. Only the tool→agent response boundary is genuinely new, so the "output-only undercounts exposure by 2×" headline reduces to "tools return more than was asked for."
  • The one new boundary measures something the agent does not control, which makes it a strange thing to score as an agent privacy violation. What a tool returns is a property of the tool's API, not of the model. The paper's own results confirm this reading: PVR moves only from ~91% to 77.5% even under block, because a guard cannot make a tool return less. So the paper's headline metric is dominated by a stage that neither the agent nor the authors' own intervention can change, and the justification for counting it (data "could be logged, cached, or recalled" on a third-party server) is asserted about MCP deployments rather than measured.
  • Scenarios are deliberately built so a legitimate query returns both the appropriate and inappropriate item, and a run counts as violated if any of four boundaries fires. The result is 83–99% for every one of 16 models, a 16-point spread that is barely above the stated 3-point standard error. A metric that cannot separate Claude Opus from Granite is not yet useful for comparing agents, which is what a benchmark is for.
Questions for authors
  1. If you exclude the boundaries that data-minimization evaluation already covers and the instruction stage that your scenario design fixes, what is left of the PVR–LR gap beyond the tool-response stage?
  2. the benchmark is 677 scenarios in the abstract and Appendix I.1 but 750 in Table 2, §6.1 and the limitations; the subset is 85 in the body and 100 in Tables 10–11 and the limitations; the user-disclosure split is given as 59/7/34, 72/5/24 and "70% fundamentalist" in three places; the abstract claims "at least 1.5×" where Table 1's minimum is 1.7 and its maximum is 2.7, against a section heading saying 1.5–2.4. Validation is described as "expert review of every scenario" in Figure 2 and §3 and then as "spot-checked any flagged scenarios... not a review of every scenario" at the end of the same section. Two cross-references are broken ("Appendix 7", "Table ??").
3 comments

We thank the reviewer for the thoughtful review. Point-by-point response below.

Concern 1: Novelty relative to AgentDAM (W1).

AgentDAM [1] is closely related work, but the two benchmarks evaluate different agents and different flows. AgentDAM tests web and computer-use agents, including Claude computer use, on three WebArena sites (Reddit, GitLab, and Shopping), where the agent acts through browser commands such as click, type, and goto instead of calling tool APIs. The sensitive data is handed to the agent up front, as a private document or conversation attached to the instruction, and the judge then checks whether that data "appears in the action text" of what the agent types or posts [1]. What AgentDAM scores, then, is what the agent sends out.

Σ-Bench evaluates a different setting, in which tool-calling agents work in the user's own apps (calendar, email, contacts, messaging), as personal agents now in deployment such as Meta's Muse [2] do. Here the agent gets nothing sensitive up front, and data enters the run only when the agent asks for it and a tool returns it. This is what lets our query boundary measure whether the agent requests more than the task needs, for example searching all of Rachel's emails when the task names one meeting. AgentDAM does not score this, because its data is already in the agent's context before the agent acts, and it does not score what the web page shows the agent after each action either.

The two benchmarks are therefore complementary, since AgentDAM covers what web agents send out and Σ-Bench covers what tool-calling agents request and receive. The comparison also exposes an inaccurate sentence in our related work, which says each prior benchmark scores "the agent's final output." Since AgentDAM checks every outbound action, we will write "the agent's outbound actions" and add the comparison above.

Concern 2: What remains of the PVR–LR gap (Q1).

Since submission, we have expanded the benchmark from 677 to 750 scenarios and the evaluation subset from 85 to 100, and every scenario now passes an automated check that its task, app data, and contact book agree. We also gave the tools the narrower filters that real APIs provide, such as selecting calendar events by date or tag, which removes the 11.1% floor that Appendix B reports for a perfect agent under the original tools, and added to every scenario an oracle trajectory that completes the task with PVR_QO = 0. With these filters no tool is forced to return an item the task does not need, so the oracle's remaining PVR of about 20% comes entirely from instructions in which the user already shares a sensitive detail the task does not need, which neither the agent nor the tools can undo.

We ran the experiments on the expanded benchmark, using the 100-scenario subset with three trials per scenario and eight of the sixteen models so far.

Most of the gap lies at the query boundary, which AgentDAM does not score (Concern 1). The gap is made up of runs whose final output is clean but that still have a violation earlier in the pipeline. In every model, more than half of these runs, between 54% and 73% and 63% on average, are ones where the agent itself asked a tool for more than the task needed. Because the oracle reaches PVR_QO = 0 on every scenario, a narrower request was available in each of these runs, so this part of the gap comes from the agent's choices, not from how the benchmark is built. Most of the rest are runs where the tool returned extra on its own, and the user's instruction accounts for a small remainder.

ModelRuns with a clean output but an earlier violation (PVR − LR)Of these, runs where the agent asked for more than needed
Claude Haiku 4.541.762%
Claude Opus 4.751.373%
GPT-5.547.061%
Granite 4.1 30B55.454%
GLM-4.6V46.862%
MiniMax M2.540.062%
Mistral Large 354.369%
Kimi K2.743.764%
Mean47.563%

This matches the split implied by the means in the submitted Table 1, where 61% of the gap comes from queries. The revision will report all 16 models, with the newer Kimi K2.7 in place of K2.5, which is no longer available to us.

Refs:

[1] Zharmagambetov, A., Guo, C., Evtimov, I., Pavlova, M., Salakhutdinov, R., & Chaudhuri, K. (2026). Agentdam: Privacy leakage evaluation for autonomous web agents. Advances in Neural Information Processing Systems, 38.

[2] Introducing Muse: The World’s First Personal AI Agent Built for Everyone https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/

Concern 3: The response boundary measures the tool, not the agent (W2).

We agree that what a tool returns depends on its API, not on the agent, so we do not use the response boundary to compare agents. Agents are compared on PVR_QO, which counts only their queries to tools and their output to the recipient, and the response boundary is reported separately as its own per-stage rate.

We keep it in PVR because privacy is a property of the whole system, and how a tool handles data matters as much as how the agent does. When Emma's calendar returns her fertility appointment for a request about her standup, an MCP calendar server sends that appointment across the network to the agent, which resends it to its model provider, all for a task that never needed it. PrivacyTrace records this transfer as an edge, so it is measured, not assumed. Whether the data is later "logged, cached, or recalled" is a further risk that we did not measure, and we will say so, but counting the transfer does not depend on it, because under data minimization (GDPR Article 5(1)(c)) sending data a task does not need is the problem in itself.

Measuring this boundary also opens a research direction that output-only evaluation cannot support, which is making the tools themselves more private, whether through privacy-preserving APIs, data minimization at the tool, or guards like ours that withhold an item before the agent reads it. Our own guard already shows how far this boundary can move. In the new runs (Concern 2) the response boundary is scored on what actually reaches the agent's context, which is the transfer described above, and blocking with the guard lowers response-boundary violations from 90% of runs to 20%, and PVR from 96% to 49%, averaged over eight models.

For the camera-ready, Section 2 will describe the response boundary as the tool's part of the pipeline, and the discussion will present tool-side privacy as a direction the benchmark supports.

Concern 4: PVR is high for every model and cannot separate agents (W3).

We agree that PVR alone does not rank agents. It measures how common pipeline exposure is, which is the paper's main finding, and this is why Table 1 also reports PVR_QO, the rate at the two boundaries the agent controls, for comparing agents. PVR_QO does separate Opus from Granite, which differ by 22 points in Table 1 and by 16 points in the new runs (Concern 2), well outside the standard error of about 3 points per model. The separation is not a side effect of agents failing tasks, because counting only runs where the task succeeds, PVR_QO still ranges from 77.6 to 89.7 across the eight models. The table below gives the new runs for the eight models run so far; the other eight will follow in the revision.

ModelLRPVR_QOPVRTSR
Claude Haiku 4.547.073.088.779.7
Claude Opus 4.748.385.799.792.0
GPT-5.552.381.099.397.0
Granite 4.1 30B39.469.294.873.0
GLM-4.6V51.079.897.783.3
MiniMax M2.559.784.399.792.0
Mistral Large 345.082.399.385.3
Kimi K2.748.376.392.077.0
Mean48.979.096.484.9

Each number is the percentage of runs, over 100 scenarios with three trials each, and the standard error is 3 to 5 points per model, as in Table 1. Absolute levels differ from Table 1 because the scenario set has changed, but every finding the reviewer asks about holds, since PVR exceeds LR for every model, Granite still has the lowest LR, and PVR_QO separates the models.

The co-location of needed and unneeded data is deliberate, but it does not force the result. Because the oracle reaches PVR_QO = 0 on every scenario, a careful agent can always avoid query and output violations, and in the new runs at least one model does so on half the scenarios, as do 15% of all runs. Most runs still do not, so these violations come from what agents choose to request and send.

For the camera-ready, Section 6 will compare agents on PVR_QO and LR, and use PVR to show how often pipeline exposure occurs across all agents.

Concern 5: Inconsistencies (Q2).

We thank the reviewer for the careful reading; all six are real. (i) the benchmark has 750 scenarios; the submitted version reported 677 because the remaining scenarios were still going through validation, and all 750 now pass it and are used. (ii) The subset has 100 scenarios; the submitted runs used the 85 that had passed validation by then, and all 100 are used in the new runs. (iii) 59/7/34 is the full benchmark, 72/5/24 the subset, and "70%" the rounded subset figure; each will be labeled. (iv) The abstract, heading, and results table will quote the same range, which on the new runs is 1.7x to 2.4x. (v) Every scenario passed rule checks and up to three LLM critique-and-revise cycles, and the authors then went through the benchmark persona by persona, reviewing every scenario the checks flagged and holding back any that did not yet pass; Figure 2 and Section 3 will describe this process and will describe the consistency check from Concern 2, which all 750 scenarios pass. (vi) "Appendix 7" is Appendix B, and "Table ??" in Appendix J is Table 2. We will fix all six in the revised paper, so that every count, range, and reference matches.

We hope these responses address the reviewer's concerns and are happy to discuss any point further during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.