- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that privacy in agentic workflows should be evaluated over the full execution pipeline rather than only the final output. The authors introduce Σ-Bench, a benchmark of privacy-sensitive personal-data scenarios, and PrivacyTrace, a graph representation that records information flows across instruction, query, tool-response, and output boundaries. They evaluate 16 agent models and find that pipeline violations are much more frequent than final-output leaks.
Critical strengths
- The paper addresses an important limitation of output-only privacy evaluation. In agentic systems, sensitive data may be exposed through tool calls, tool responses, intermediate context, or logs even when the final response looks safe.
- The empirical study is broad, covering many frontier and open-weight models, multiple privacy domains, and several intervention settings.
Critical weaknesses
-
The paper uses a conservative rule that a transfer is appropriate only if it is both needed for the task and fitting for the recipient. This makes sense as a strict auditing stance, but the paper often presents every upstream exposure as a privacy violation without sufficiently distinguishing trusted local processing, tool-side over-return, and actual disclosure to an external recipient.
For example, in the opening sick-leave scenario, saying “out sick” is itself health-related information requested by the user to be sent to the manager, while fertility and therapy appointments are clearly more sensitive and not task-relevant.
The paper would be stronger if it clarified this distinction rather than treating the example mainly as a binary private/not-private contrast.
-
The benchmark relies heavily on synthetic personas, synthetic scenarios, and LLM-assisted scenario generation and judging. The authors report validation steps and cross-judge checks, but there is no full human annotation study. Since contextual privacy norms are subtle and scenario-dependent, this weakens confidence in the ground truth labels and in claims about what is “appropriate” across domains.
-
PrivacyTraceGuard has limited practical effectiveness under the paper’s own metric. Even with filtering or blocking, pipeline violation rates remain around 80% and 77.5%, respectively. This raises a concern that the proposed intervention mostly shifts where leakage is observed rather than providing a robust solution to contextual privacy in agentic workflows.
Questions for authors
Please refer to the Weaknesses.
3 comments
Concern 3: The labels and judge rely on LLMs, without a full human study (W2).
Σ-Bench is synthetic, as are PrivacyLens [1] and CIMemories [2], because real user data with privacy labels would itself be risky to collect and share. Since the labels are synthetic too, we check them in three ways. First, every scenario passes structural rule checks, LLM critique, and an automated check that its task, app data, and contact book agree, and all 750 scenarios of the expanded benchmark described below pass; the authors also went through the benchmark persona by persona, reviewing every scenario the checks flagged and holding back any that did not yet pass. Second, a rule-based evaluator that flags only exact ground-truth items, and so cannot over-flag, still finds PVR more than twice LR (68.0 against 30.7 on the new runs, and 65.2 against 26.0 in Appendix I.6). Third, a proprietary judge and an open-weights one, gpt-5.5 and gpt-oss-120b, agree with Cohen's κ between 0.63 and 0.88 across the four boundaries, after raising the open-weights judge's output-length limit, which in the submission was set too low for a reasoning model and cut its answers short.
Since submission, we have expanded the benchmark from 677 to 750 scenarios and the evaluation subset from 85 to 100, and every scenario now passes an automated check that its task, app data, and contact book agree. We also gave the tools the narrower filters that real APIs provide, such as selecting calendar events by date or tag, which removes the 11.1% floor that Appendix B reports for a perfect agent under the original tools, and added to every scenario an oracle trajectory that completes the task with PVR_QO = 0. With these filters no tool is forced to return an item the task does not need, so the oracle's remaining PVR of about 20% comes entirely from instructions in which the user already shares a sensitive detail the task does not need, which neither the agent nor the tools can undo.
We ran the experiments on the expanded benchmark, using the 100-scenario subset with three trials per scenario and eight of the sixteen models so far; the other eight will follow in the revision.
| Model | LR | PVR_QO | PVR | TSR |
|---|---|---|---|---|
| Claude Haiku 4.5 | 47.0 | 73.0 | 88.7 | 79.7 |
| Claude Opus 4.7 | 48.3 | 85.7 | 99.7 | 92.0 |
| GPT-5.5 | 52.3 | 81.0 | 99.3 | 97.0 |
| Granite 4.1 30B | 39.4 | 69.2 | 94.8 | 73.0 |
| GLM-4.6V | 51.0 | 79.8 | 97.7 | 83.3 |
| MiniMax M2.5 | 59.7 | 84.3 | 99.7 | 92.0 |
| Mistral Large 3 | 45.0 | 82.3 | 99.3 | 85.3 |
| Kimi K2.7 | 48.3 | 76.3 | 92.0 | 77.0 |
| Mean | 48.9 | 79.0 | 96.4 | 84.9 |
Each number is the percentage of runs, over 100 scenarios with three trials each, and the standard error is 3 to 5 points per model, as in Table 1. Absolute levels differ from Table 1 because the scenario set has changed, but the paper's main findings hold on the new runs. PVR is 1.7 to 2.4 times LR for every model, and PVR_QO, which counts only the boundaries the agent controls, is still about 1.6 times LR and still separates the models. The response boundary is still where violations are most frequent, and on the four models with prompting results so far, a privacy-aware prompt does not lower LR. Kimi K2.7 replaces K2.5, which is no longer available to us.
We also ran a human study for this rebuttal. Two annotators independently labeled the same 50 pipeline steps, sampled from the submitted runs of all sixteen models and covering all four boundaries (11 to 16 steps each), without seeing the judge's verdicts. They agree with each other with Cohen's κ of 0.72, so the labels people give are consistent. With both applying the same written rule at each boundary, they agree with the judge with an average κ of 0.73 at the instruction boundary, 0.42 at response, and 0.82 at output. At the query boundary agreement is lower (κ 0.23), and in all 11 disagreements there, the annotators flagged a query that the judge allowed. The query rates we report therefore understate, if anything, how often agents ask for more than they need. The revision will extend the study to more steps per boundary.
Refs.
[1] Shao, Y., Li, T., Shi, W., Liu, Y., & Yang, D. (2024). Privacylens: Evaluating privacy norm awareness of language models in action. Advances in Neural Information Processing Systems, 37, 89373-89407.
[2] Mireshghallah, N., Mangaokar, N., Kokhlikyan, N., Zharmagambetov, A., Zaheer, M., Mahloujifar, S., & Chaudhuri, K. (2026, April). CIMemories: A compositional benchmark for contextual integrity in LLMs. In International Conference on Learning Representations (Vol. 2026, pp. 95513-95532).
Concern 4: The guard only shifts where leakage appears (W3).
If the guard only shifted leakage, the total would stay the same, but it drops instead, at every boundary. The table below gives the new runs for the eight models, on the expanded benchmark described in Concern 3.
| Mean of eight models | Instruction | Query | Response | Output (LR) | PVR_QO | PVR | TSR |
|---|---|---|---|---|---|---|---|
| No guard | 19.4 | 60.8 | 90.3 | 48.9 | 79.0 | 96.4 | 84.9 |
| Guard, filter | 2.1 | 40.9 | 56.8 | 12.2 | 47.2 | 75.9 | 66.2 |
| Guard, block | 1.0 | 39.2 | 20.0 | 1.8 | 40.0 | 48.5 | 28.2 |
Violations fall at every boundary, for every model, so what the guard removes at one boundary does not reappear at another. Response violations fall too, because the new runs count only what reaches the agent, and the guard stops a withheld return from reaching it. The cost is task success, as in the submitted Table 2. Most of the drop comes from what the guard withholds. When it holds back a request or a tool's return, the agent often cannot finish, and under block a flagged instruction is refused outright, which ends about one run in five. Filter therefore keeps two thirds of tasks working while cutting output leaks by about three quarters, and block brings PVR below 50 at a larger cost, which is why we recommend filter as the default and block for settings where a leak matters more than finishing the task. The guard also turns leaky completions into clean ones. On the 24 of the 100 scenarios where every unguarded run that completes the task also leaks in its output, a guarded run completes the task without the leak on 22. For the camera-ready, we will report these results for all sixteen models and lead the guard results with LR and PVR_QO.
We hope these responses address the reviewer's concerns and are happy to discuss any point further during the discussion period.
We thank the reviewer for the review, and for recognizing the breadth of the study across models, privacy domains, and interventions. Point-by-point response below.
Concern 1: Upstream exposure is not separated from disclosure to an external recipient (W1a, W1b).
We agree these are different, and the four boundaries keep them apart. External disclosure is the output boundary, which LR measures, and tool over-return is the response boundary. Local processing is whatever lands inside the agent, namely the user's instruction and the tool's response, whereas the query lands in the tool and the output lands with the recipient. Because each boundary is scored on its own, every trust stance a deployer might take maps to a metric we report, and the table below computes all four from the same runs (new runs on the expanded benchmark described in Concern 3, mean of eight models).
Trusting the tools alone changes little, because what they return still enters an untrusted agent, while trusting the agent lowers the rate the most. Even with both trusted, about half of runs send an unneeded item to an external recipient, so the main finding does not rest on the zero-trust default.
This flexibility is part of the design, because a deployer chooses which boundaries count for their setting, and the guard can be set separately at each one to allow, filter, or block. A deployer who hosts the agent themselves, for example, can treat what reaches the agent as safe and check only what the agent sends out, its queries to tools and its messages, which is exactly PVR_QO. For the camera-ready, Section 2 will present this table, with the zero-trust default as one of the four rows, and will explain how a deployer sets the boundaries to match their trust zone.
Concern 2: The sick-leave example (W1c, W1d).
The reviewer is right that being out sick is health information, so our sentence saying the instruction "contains no sensitive information" was wrong. The instruction is still appropriate, because the task needs it and the manager is the right person to receive it, which is not true of the fertility and therapy appointments. We will rewrite the sentence and relabel the instruction edge in Figure 1 as appropriate because it is needed and fits the recipient, so that the example applies our rule instead of appearing to ignore it.