- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AgentCIScope evaluates contextual integrity in LLM agents through executable tasks across multiple applications. Its construction pipeline combines privacy contracts, a fixed attribute taxonomy, LLM persona agreement, policy validation and stress-test instruction generation. The benchmark contain tasks that derived from contracts across environments. agent evaluation over trajectories reveals substantial leakage, differences across agent stacks and information types and additional failures under repeated execution.
Critical strengths
- Broad executable evaluation.
The benchmark supports multistep information flows across diverse applications and evaluates several model framework combinations. It provide potentially useful extension beyond privacy judgments and isolated actions and demonstrates that task completion could coexist with substantial disclosure failures.
- Structured construction with human validation.
Explicit recipient-specific contracts, coherent environment population and evidence-based leakage judgments makes the evaluation auditable. Appendix D’s human agreement result provide encouraging support for the generated labels and automated judge.
Critical weaknesses
- Surrogate-based selection limits the interpretation of the results.
Every retained task must induce leakage from GPT-5.6 Sol with OpenClaw 3.3. It is reasonable for constructing a stress test, but the reported rates characterize a deliberately selected failure set. Without results on base instructions, unfiltered candidates, and alternative surrogates, it remains unclear how much the observed agent rankings and attribute or environment patterns depend in the selection procedure. Moreover, the tasks share only underlying contracts, whereas the reported Wilson intervals don't account for this dependence.
- The policy-grounding claim needs stronger validation.
Contract generation, persona judgments, policy validation and leakage evaluation use the same model. So persona agreement don't establish independent normative agreement. Human validation is encouraging, but covers only tasks from agent configuration, and assesses attribute privacy status rather than reporting a separate audit of policy applicability. Because agents receives neither contracts nor policies, the benchmark also needs to establish that relevant recipient permissions, and disclosure restrictions are inferable from the visible context. Otherwise, some failure might reflect unavailable contextual information rather than clear violations.
Questions for authors
-
How does leakage rate and agent ranking change on base instructions, and the unfiltered candidate pool? What fraction of candidates survives surrogate filtering?
-
What information visible to tested agent establishes recipient authorization and disclosure restrictions? Does providing applicable policy context materially change the result?
-
How many human annotator participated, and are they blinded to the generated labels? Is policy applicability independently assessed and does judge reliability hold across other agent configurations?
-
Do the framework differences remain significant under paired analysis with uncertainty estimated in the contract level? Are system prompts, reasoning budgets, decoding setting and tool permissions controlled and documented?
-
How sensitive is the reported leakage rate to explicit CI reasoning, given prior evidence that reasoning and RL post-training improve contextual integrity and preserving task performance [1]?
[1] Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. NeurIPS 2025.
5 comments
W2-A. Additional Validation of Persona Judgments, Policy Applicability, and LLM Judges (Appendix G)
Thank you for this valuable suggestion. To strengthen the independent validation of our benchmark, we conducted additional experiments examining persona judgments, policy applicability, and leakage evaluation.
1. Persona Judgment Validation. To assess the reliability of persona judgments across different model families, we independently re-evaluated all attributes using three models: GPT-5.4, Claude Sonnet 5.5, and Gemini 3.1 Pro. The results show consistently high inter-model agreement:
| Model Pair | Agreement | Cohen's κ [95% CI] |
|---|---|---|
| GPT-5.4 vs. Claude Sonnet 5.5 | 96.2% | 0.93 [0.91, 0.95] |
| GPT-5.4 vs. Gemini 3.1 Pro | 94.0% | 0.89 [0.86, 0.91] |
| Claude Sonnet 5.5 vs. Gemini 3.1 Pro | 93.9% | 0.89 [0.86, 0.91] |
These results demonstrate strong consistency in persona judgments across different model families, reducing concerns about reliance on a single model's judgments.
2. Policy Applicability Validation.
To validate the applicability of the policies in our benchmark attributes, we conducted both cross-model validation and additional human annotation.
Cross-Model Validation. We evaluated all attributes using three models: GPT-5.4, Claude Sonnet 5.5, and Gemini 3.1 Pro. The results are summarized below.
Table 1. Policy Applicability Validation
| Model | Applicable | Not Applicable | Uncertain |
|---|---|---|---|
| GPT-5.4 | 90.4% | 2.9% | 6.7% |
| Claude Sonnet 5.5 | 85.3% | 5.0% | 9.7% |
| Gemini 3.1 Pro | 93.2% | 5.0% | 1.8% |
Table 2. Inter-Model Agreement on Policy Applicability
| Model Pair | Agreement Rate |
|---|---|
| GPT-5.4 vs. Claude Sonnet 5.5 | 86.8% |
| GPT-5.4 vs. Gemini 3.1 Pro | 93.0% |
| Claude Sonnet 5.5 vs. Gemini 3.1 Pro | 89.5% |
All three models judged over 85% of the policy-attribute pairs as applicable, with pairwise agreement rates ranging from 86.8% to 93.0%.
Human Validation. To further validate policy applicability independently of LLM judgments, three human annotators assess 30 sampled cases, covering 167 policy applicability judgments. All three annotators unanimously judged 140 out of 167 (83.8%) as applicable.
These results provide additional evidence of policy applicability across independent model and human assessments, strengthening the reliability of our policy-grounded benchmark.
3. Leakage Evaluation Validation. To assess the reliability of our LLM-based leakage evaluation, we conducted additional validation using both independent model judgments and human annotations.
3.1. Human validation across agent configurations. Three annotators independently labeled 150 agent trajectories, covering the same 30 tasks under five configurations spanning all four frameworks. For each protected attribute specified in the privacy contract (460 attribute instances in total), annotators determined whether it was leaked to a forbidden recipient. We used majority voting to establish the human reference labels. The table below reports the judge's agreement with these labels at both the run and attribute levels (run-level / attribute-level).
| Configuration | Judge–Human Majority Agreement (Run / Attribute) |
|---|---|
| Claude Code + Claude Opus 5 | 96.7% / 96.7% |
| Google ADK + Gemini 3.8 Flash | 100% / 98.9% |
| OpenAI Agents SDK + GPT-6 Astra | 96.7% / 94.6% |
| OpenAI Agents SDK + Grok 4.6 | 100% / 94.6% |
| OpenClaw + Kimi K3 | 96.7% / 97.8% |
| Overall | 98.0% / 96.5% |
Overall, these results demonstrate that the judge's reliability extends beyond GPT-6 Astra with the OpenAI Agents SDK. We believe our evidence-based leakage assessment, which grounds judgments in concrete evidence from agent trajectories, also helps support the consistency and reliability of these evaluations.
3.2. Cross-model judge consistency. We further evaluated 600 trajectories (50 tasks × 12 agents) using two additional LLM judges, Gemini 3.8 Flash and Claude Haiku 5.5, with the same evaluation prompt as our primary GPT-5.4 judge. Compared with GPT-5.4, Gemini 3.8 Flash achieves 90.7% run-level agreement, while Claude Haiku 5.5 achieves 91.7%. At the attribute level, the two pairwise comparisons with GPT-5.4 yield agreement rates of 94.7% and 96.0%. Importantly, the relative ranking of all 12 agents by leakage rate remains largely consistent across judges. These findings further support the robustness of our evaluation conclusions across different judge models.
W2-B and Q2-A: Agent Context Visibility and Authorization
Thank you for raising this important point. We would like to clarify that agents receive user instructions specifying the intended recipient and task purpose. Our environment construction also ensures that the recipient's role is reflected in the environment data. Agents are expected to assess whether disclosing sensitive information to the recipient is appropriate based on the available context. Additionally, our quality gate ensures that user instructions do not explicitly request sensitive information.
Regarding the concern that some failures may reflect insufficient contextual information rather than clear violations, we argue that uncertainty about authorization should not justify disclosing sensitive information. When the available context does not clearly establish whether disclosure is authorized, a privacy-conscious agent should not assume permission. We believe defaulting to non-disclosure under uncertainty is an important principle of responsible agent behavior.
Q2-B / Q5: Impact of Policy Context and Explicit CI Reasoning (Appendix B, Table 4)
Thank you for these valuable suggestions. To investigate whether providing explicit policy context and CI reasoning guidance reduces privacy leakage, we conducted additional experiments on the 100 tasks sampled from 100 distinct CI contracts. We evaluated three agents under the same tasks, environments, and judges. The full results are provided in the Appendix.
| Defense | Kimi K3 (OpenClaw) | Gemini 3.8 Flash ( Google ADK) | GPT-6.1 Sol (OpenAI Agents SDK) |
|---|---|---|---|
| No defense (TLR) | 92% | 95% | 47% |
| Explicit policy context | 79% (−13 pp) | 63% (−32 pp) | 17% (−30 pp) |
| Privacy-enhancing prompt (PrivacyLens; Shao et al., 2024) | 90% (−2 pp) | 78% (−17 pp) | 31% (−16 pp) |
| CI-CoT prompting (Lan et al., 2025) | 69% (−23 pp) | 58% (−37 pp) | 16% (−31 pp) |
| Data-minimization prompt (AgentCIBench; Goel & Gurevych, 2026) | 91% (−1 pp) | 82% (−13 pp) | 36% (−11 pp) |
| CI-based supervisor (Ghalebikesabi et al., 2024) | 70% (−22 pp) | 78% (−17 pp) | 30% (−17 pp) |
The results show that both providing explicit policy context and incorporating CI reasoning reduce privacy leakage across all three agents. However, privacy leakage persists despite these interventions, demonstrating that AgentCIScope remains effective in identifying and distinguishing agent vulnerabilities even with additional privacy safeguards.
Q3: Human Validation (Appendix G)
In our revised version, three human annotators participated in the validation and were blinded to the generated labels. We also conducted an independent assessment of policy applicability and extended our judge reliability evaluation to additional agent configurations.
Q4-A: Statistical significance of framework differences (Appendix E, Table 8)
Thank you for this suggestion. We conducted a paired analysis using contract-level cluster bootstrapping to compute 95% confidence intervals for framework differences. Overall, the results suggest that Claude Code and OpenClaw tend to exhibit lower privacy leakage rates than OpenAI Agents SDK and Google ADK. With the backbone held fixed, the framework difference remains significant for Claude Opus 5 (Claude Code vs. OpenClaw: −11.1 pp, 95% CI [−14.9, −7.3]), GPT-6 Astra (OpenClaw vs. OpenAI Agents SDK: −8.1 pp, [−10.2, −6.1]), and Kimi K3 (OpenClaw vs. Google ADK: −4.2 pp, [−6.7, −1.8]).
Q4-B: Agent Configuration Details (Appendix H)
Thank you for this suggestion. We have expanded the Appendix to provide more detailed documentation of system prompts, reasoning budgets, decoding settings, and tool permissions across agent configurations.
Thank you again for your thoughtful review and constructive feedback. We hope the additional experiments and clarifications address your concerns. We would be grateful if you would reconsider your assessment in light of these revisions. Thank you for your time and effort!
Thank you for recognizing the breadth of our executable evaluation, the structured benchmark construction, and the supporting human validation. Below, we address each weakness and question.
Thank you for raising this concern. Across all generation runs, our pipeline evaluated 3,420 candidate instructions against the surrogate agent and retained 936 (27.4%) that triggered leakage.
Regarding the motivation for surrogate-based filtering, our goal is to construct a challenging benchmark that helps the community identify privacy risks and advance privacy-preserving agents. We believe selecting challenging cases provides greater value for stress-testing existing agents and driving future improvements.
Regarding the impact of filtering on agent rankings and attribute/environment patterns, we acknowledge that our current findings are based on the filtered benchmark. Due to time and API cost constraints, evaluating all agents on the full unfiltered candidate pool is impractical. Instead, we conducted an additional experiment on 80 tasks from 80 different contracts, evaluating eight agents with both benchmark instructions (i.e., the filtered instructions) and randomly sampled filtered-out candidates (i.e., instructions on which the surrogate did not leak). We did not separately evaluate base instructions, as they serve as intermediate inputs to our instruction-generation pipeline rather than finalized evaluation tasks. Both conditions used the same contracts, environments, and judges, differing only in the instructions. Although leakage rates were lower on filtered-out instructions, the relative agent rankings remained highly consistent (Kendall's τ = 0.93), suggesting that selection does not substantially affect cross-agent comparisons in this evaluation.
Thank you for raising this. The original Wilson intervals treat tasks as independent. To account for within-contract dependence, we additionally computed 95% confidence intervals using a contract-level cluster bootstrap, resampling contracts rather than individual tasks. The resulting intervals are 1.2-1.9× wider than the Wilson intervals, but our main conclusions remain unchanged: low- and high-leakage agents remain clearly separated, and agent rankings remain stable. We have added the methodology and results to the revised paper.