acceptodds
Under review as a conference paper at ICLR 2027

PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents

Abstract

While prior work has documented privacy failures in LLM agents, it remains unclear how the presentation of privacy guidance influences their choice of information sources. We introduce PrivacySkills, a controlled framework for evaluating how agents choose among acquisition pathways that provide the same task-relevant value: consulting publicly available personal information, accessing confidential sources, or interacting with the user. The evaluation framework comprises 55 synthetic tasks spanning 11 categories of personal information, with 169 associated skills that describe the available acquisition pathways. We consider privacy guidance through system-level instructions, skill-level metadata labels, or both. Separately, we vary user availability and urgency framing. With users available and no privacy guidance, agents access confidential sources in 30% of valid runs on average across five open-weight models, despite sufficient alternatives. This rate increases to 45% when users are unavailable, whereas urgency framing has no detectable effect. System-level privacy instructions alone have limited effects on confidential access, while skill-level intrusiveness labels produce a modest reduction (24% on average), but combining the two roughly halves confidential access. Our findings motivate incorporating privacy annotations into skill specifications and evaluating their effectiveness alongside system-level instructions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Privacy, Tool Use, Agent Skills, Privacy Benchmark, Source Selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fE37OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how LLMs choose information sources that vary in sensitivity to complete tasks. To evaluate this question, an evaluation framework is developed by synthetically generating information sources as available skills, personal information categories, and tasks. The experimental results show that LLMs access confidential sources 30% on average, and including an intrusiveness metadata to the skills improve the intrusion rate.

Critical strengths

I think that the paper has some good strengths. The first main one is that the experimental design seems mostly solid. I like that the personal information categories are based on taxonomies from real-world legislation (GDPR, CCPA, LGPD, PIPL). Implementing the capabilities as skills following contemporary agent harnesses aligns well with the recent trend towards agentic frameworks. And controlling for any variation via deterministic outputs from the skills. The description of the experimental design is also well-detailed and documented, which can help with reproducibility.

Moreover, the paper addresses a timely problem of better understanding how LLMs make information-sharing decisions. The specific research questions explored are novel. To my knowledge, I don’t know of any works that have evaluated whether LLMs understand data source sensitivity.

Lastly, I like the results about the intrusiveness metadata as it well motivates the need for incorporating privacy annotations into skill specifications. I believe that this is a critical requirement for designing guardrails for agents, which nicely connects with the results from this paper.

Critical weaknesses

I have some reservations about the papers that lead me to weakly reject the work, but I would be open to changing my score if these reservations are clarified.

The first main reservation may be too open-ended to address, but the significance of the work feels a bit shaky to me. While I do agree that evaluating the problem of how LLMs choose data sources with varying sensitivity is interesting, I’m not entirely convinced that the problem and the insights derived from the results significantly contribute to the existing AI privacy work, at least to meet the ICLR significance bar. But I do think that the evaluation framework proposed in this work is a good contribution.

The second main reservation I have is the models evaluated in the work. Since they are all open-source models, I’m unsure if the results obtained from evaluations are limited only to open-source models. For example, it could be the case that frontier models rely on confidential sources significantly less than open-source models. In that case, the insights from the current results may not hold for more capable models. It would be extremely useful to include results from models such as the recent versions of GPT or Sonnet.

Lastly, I had a few minor critiques.

  • For RQ 2, it would be more insightful to include the Acquisition tiers distribution per model. In particular, does the intrusiveness metadata just cause the LLMs to ask the user more?
  • It would also be insightful to include an example containing the task, the specific user, public, and confidential source skills in the main text.
  • There’s a lot of emphasis on valid runs through the main text. It would help to supplement Figure 3 by also including the number of total runs and valid runs together with each model to get a sense of the success rate in the context of source selection.
Questions for authors
  • How many runs were attempted for each model in Figure 3? And how many of them were valid? Could you also explain to me the connection between Table 1 in Appendix A and the main results? Did each model get a different number of attempted runs?
  • Did you do an error analysis for the sources of why certain runs would be invalid?
5 comments

We thank you for your careful reading of our paper and for your constructive feedback. We have also updated the attached manuscript. We address the concerns and questions raised below:

Critical Weaknesses: Significance. Source choice is important for the privacy protection later on, because it determines which personal data (e.g., a private medical document vs. a public profile) enters the context of the LLM, in order to extract a same missing attribute to resolve a task. Our experiments identify two factors that affect this choice. First, confidential accesses are performed even when a public source could be used and the user is available, and it increases when the user is unavailable. Second, a privacy instruction alone is not sufficient to remediate the problem but combining privacy instruction with skill labels provides better results. These findings foster incorporating privacy annotations into skill specifications and evaluating their effectiveness.

Critical Weaknesses: Proprietary models. Thank you for this suggestion. We added Appendix F (lines 1543-1619), which evaluates five proprietary LLMs on our benchmark: GPT-5, GPT-5.4 mini, GPT-5.4 nano, GPT-6 Astra, and Claude Sonnet 5.5. Due to cost and time constraints, these evaluations use 3 valid runs per task, setting, and tool permutation, rather than the 10 valid runs used for open-weight models. Our findings largely carry over:

  1. Proprietary models still access confidential sources in a substantial share of runs. Their HIR ranges from 12% to 31% (22.6% on average), compared with 10% to 45% (30% on average) for open-weight models.
  2. User unavailability raises HIR for all five models (38% on average). As for open-weight models, baseline user runs are redirected to the confidential source more often than each model’s baseline preference for confidential over public sources would predict. Unlike open-weight models, however, differences between proprietary models do not narrow.
  3. The response to privacy guidance varies across models. GPT-5, GPT-6 Astra, and Claude Sonnet 5.5 respond strongly to either form of guidance. GPT-5.4 mini and nano are barely affected by the instruction alone, and for them, combining instruction and labels exceeds the sum of the separate effects, as for open-weight models.

Critical Weaknesses: Acquisition tiers distribution per model. We now present in Appendix G and in Fig. 11 (lines 1620-1652) how the acquisition tiers are distributed with and without guidance, for each model. On average, the privacy instruction alone barely reduces HIR (−1.1 pp, from 30.0% to 28.9%), labels alone yield a moderate reduction (−7.3 pp, i.e., −24% in relative terms), and combining both roughly halves it (−15.1 pp, to 15%), with the largest reduction for every model. This reduction does not merely shift runs toward asking the user: under label + instruction, the public source absorbs most of it (+12 pp on average), while user runs increase by only 3 pp.

Critical Weaknesses: Example. We have added a detailed example in Section 3.1 and Fig. 3 (lines 211-224), containing the task, the specific user, public, and confidential source skills. This makes explicit that all three pathways provide the same required value, while confidential access exposes additional personal information.

Critical Weaknesses: valid runs and Fig. 3. Thank you for this suggestion. We have supplemented Fig. 4 (the old Fig. 3) with the validity rate (lines 378-391). We have also extended Appendix A with a complete analysis, and we now provide the rate of valid and successful runs in our experiments for each model (lines 737-796).

Q1: attempts and Valid runs. For each (model, setting) pair, we run each of the 55 tasks under each of the 6 presentation orders for the skills until we obtain 10 valid runs. This gives 3,300 valid runs per cell. Consequently, models and settings with lower validity rates require more attempts (e.g., Mistral needs more attempts). This is now detailed in the complement provided in Fig. 4 (lines 378-391) and the extended Appendix A (lines 737-796). Table 1 pools all settings for each model. We found that the previous version was produced from an intermediate snapshot of the logs. We apologize for the inconsistency. The main results are unaffected by this reporting error. We have corrected Tables 1 and 2.

Q2: Error analysis. We also extend Tables 1 and 2 in Appendix A (lines 736–796) with a comprehensive error analysis by failure type:

  • (i) no acquisition (failure of missing attribute acquisition),
  • (ii) acq., no execution (the execution skill is not executed, after acquisition was performed),
  • (iii) acq., execution fails (the execution skill is called but fails), and
  • (iv) other failures (typically runs interrupted by the context-length limit).

The absence of acquisition accounts for the vast majority of invalid runs, representing 15.2 of the 17.2 pp of invalid runs.

Reviewer_fE37OpenReview ↗

Response by Reviewer

I thank the authors for their rebuttal. It has clarified most of the concerns that I had in my initial review.

However, I still have some concerns based on the results from the proprietary models. Seemingly, providing a simple instruction to "protect the user’s privacy and prefer the least intrusive source sufficient for the task" reduces the HIR to near zero for GPT-5, GPT-6 Astra, and Sonnet 5.5. Hence, this suggests that the benchmark might be too easy for frontier models, whereas providing this instruction for open-weight models had little effect. I understand that the simplified design of the benchmark allows for precise evaluation and attribution of information source sensitivity, but, as pointed out by reviewer XiVk, this simplification may be too unrealistic without enough complexity for frontier models.

AuthorsOpenReview ↗

Thanks for the follow-up discussion

We thank the reviewer for the thoughtful follow-up. We agree that a simple privacy instruction brings HIR close to zero for GPT-5, GPT-6 Astra, and Claude Sonnet 5.5 in our controlled setting. This is an encouraging result, but we believe it refines rather than undermines the findings of our study.

First, regarding RQ1 (propensity to access confidential sources): Without explicit privacy guidance, GPT-5, GPT-6 Astra, and Claude Sonnet 5.5 still access confidential sources in 27%, 12%, and 21% of valid runs, respectively, despite sufficient public alternatives. These rates increase to 37%, 27%, and 24% when the user is unavailable. Thus, even frontier models do not systematically avoid unnecessary confidential access without guidance. This also addresses the possibility raised in your initial review that frontier models might rely substantially less on confidential sources than open-weight models: our results show that this is not consistently the case.

Second, regarding RQ2 (impact of privacy guidance): The same instruction has little effect on GPT-5.4 mini (31% to 29% HIR), GPT-5.4 nano (22% to 19%), and the five open-weight models (30% to 29% on average). For these models, explicit source-intrusiveness labels provide additional improvements. Combining labels and instructions reduces HIR to 16% for mini, 3% for nano, and 15% on average for open-weight models. Our proprietary-model evaluation therefore provides a more nuanced conclusion: a simple instruction can be sufficient for certain frontier models in this setting, but its effectiveness is strongly model-dependent. This distinction is practically relevant, as smaller models also serve cost- and latency-sensitive deployments.

Finally, regarding task complexity: PrivacySkills is not primarily intended as a model-ranking challenge, but as a controlled framework for studying information acquisition decisions. By keeping the task-relevant information and available acquisition pathways fixed, we can isolate agents' propensity to access confidential sources and measure how this behavior changes with privacy guidance. This design remains informative even when some models achieve near-zero HIR under explicit instructions. We agree, however, that our findings do not establish whether these instructions remain sufficient in more complex, multi-step scenarios involving competing sources and contextual constraints. This is an important direction for future work, and we will make this limitation and the refined interpretation of RQ2 more explicit.

We thank the reviewer again for the constructive discussion and for acknowledging that most of the initial concerns have been clarified. We hope that the additional experiments on proprietary models, together with our clarification of the study's objectives and scope, address the remaining concerns about the significance of our findings.

Reviewer_o9BuOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how agents may use acquisition skills to acquire information from different sources/pathways: using public information, user interaction, or accessing confidential resources. They therefore introduce an evaluation framework with a rich set of contexts, settings, tasks, and information types and evaluate multiple models against them, providing insights into an agent's data acquisition behavior.

Critical strengths
  • [S1] This is a very timely problem that tackles a question that related literature does not: instead of solely studying what an agent chooses to reveal, the authors study how an agent decides to acquire information, which has implications for privacy outcomes downstream. The provided evaluation across several settings/contexts, tasks, . As such their related work discussion is well-described and motivates and scopes their novelty carefully.
  • [S2] Very well-executed experimental design and empirical analysis. The presentation of their framework and artifacts therein, the design choices, norm derivation from prevalent legal frameworks, and the outcomes are clear. The results are in-depth and provide useful, convincing takeaways over multiple models and different levels of privacy guidance/awareness. Indeed, evaluation under situational pressure and privacy guidance provides studies on additional realistic settings.

Auxiliary Points

I also applaud the authors for a well-thought-out work that makes careful additional considerations that were missing from the literature. In particular, I find behavioral profiling as a data category to be such an interesting consideration. The study of privacy guidance is well-presented in an ablation-esque manner, providing well-substantiated guidance on how to provide privacy guidance, and the results on user interaction absence and the subsequent evening out of differences between models is valuable as a takeaway.

Critical weaknesses
  • [W1] While the metrics the authors adopt are well-motivated and -defined, the study lacks an analysis of task completion rates and of successful transmission/acquisition of contextually appropriate attributes.

That said, while addressing this weakness will benefit the paper and indeed earn a stronger championing of it from me, the paper in its current form is valuable to the contextual integrity of AI agents community and worth presenting at the conference.

Questions for authors

Apart from seeking clarifications for the raised weaknesses, a few additional (relatively minor) concerns remain.

  • [Q1] In lines 300-301, the authors mention counterbalancing permutations of acquisition skills to counter sensitivity of tool choice to tool permutation. Does that imply that the authors ran each permutation as a separate run?
  • [Q2] Can the authors also acknowledge a few pieces of missing privacy guidance and enforcement works to provide a fuller picture of the prior literature, such as [1], [2]. and [3], which describe supervisors for contextual integrity enforcement, firewalls for input, output, and data filtering, and activation-based guardrails for input filtering, all in the area of contextual privacy for agents?

References

  • [1] Ghalebikesabi, S., Bagdasaryan, E., Yi, R., Yona, I., Shumailov, I., Pappu, A., Shi, C., Weidinger, L., Stanforth, R., Berrada, L., Kohli, P., Huang, P., & Balle, B. (2024). Operationalizing Contextual Integrity in Privacy-Conscious Assistants. ArXiv, abs/2408.02373.
  • [2] Abdelnabi, S., Gomaa, A., Bagdasarian, E., Kristensson, P.O., & Shokri, R. (2025). Firewalls to Secure Dynamic LLM Agentic Networks. ArXiv, abs/2502.01822.
  • [3] Das, S., & Fioretto, F. (2026). NeuroFilter: Privacy Guardrails for Conversational LLM Agents. ArXiv, abs/2601.14660.
2 comments

We thank the reviewer for their positive assessment of the research question, its positioning in the literature, and the experimental design. We have updated the attached manuscript.

W1. Thank you for this suggestion. We extended Appendix A (Tables 1 and 2, lines 710-796) to report, for each model and condition, the validity rate (percentage of runs that end without failure) and the success rate (percentage of runs that end with the expected effect). The more precise definitions are given in the appendix.

Over all attempts, about 90% of valid runs are successful (from 79.1% for Mistral Small 3.2 to 99.6% for Gemma 4). Importantly, task success remains similar with privacy guidance (76.7% with neutral condition, and 76.4% with label + instruction guidance).

We also break down task failure by cause:

  • (i) no acquisition (failure of missing attribute acquisition),
  • (ii) acq., no execution (the execution skill is not executed, after acquisition was performed),
  • (iii) acq., execution fails (the execution skill is called but fails), and
  • (iv) other failures (mainly runs interrupted by the context-length limit).

The absence of acquisition dominates, accounting for 15.2 of the 17.2 pp of invalid runs. Validity rates are now also reported for each model and condition in Figure 4 (lines 378-391).

Q1. Indeed, each of the 6 orders of the three acquisition skills is run separately, with 10 independent valid runs per (task, order), i.e., 55 × 6 × 10 = 3,300 valid runs per model and setting (Sec. 4 and App. D).

Q2. Thank you for suggesting these relevant references. We now discuss [1] and [3] in the “Privacy guidance and enforcement” paragraph of the related work (lines 135-138), and [2] in both the “Data minimization and information acquisition” (lines 121-122) and “Tool selection and task context” (lines 153-154) paragraphs, reflecting its dual firewall design. These works enforce contextual integrity on information that is disclosed, received, or made accessible to the agent. They are complementary to our study of the agent’s own choice of source.

Reviewer_o9BuOpenReview ↗

Response to the authors

I thank the authors for their detailed response. I do believe that the task completion rate results make this paper more holistic and stronger, and the clarifications provided with regard to my questions are satisfactory.

I retain my score, which is the highest possible, 4, and will champion this work for acceptance. Being familiar with this area and convinced of the quality of this work, I believe this will be a valuable contribution to the target community.

I wish the authors all the best and remain willing to engage in any further discussion, if the authors and/or other members of the reviewing committee so desire.

Reviewer_XiVkOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PrivacySkills, a benchmark for evaluating privacy-sensitive source selection in LLM agents. The benchmark consists of 55 synthetic tasks across 11 personal information categories and 169 executable skills, where agents must obtain the same task-relevant information by interacting with users, consulting public sources, or accessing confidential sources. To validate the benchmark, the authors conduct empirical evaluations on five open-weight LLMs, examining how source selection is influenced by user availability, urgency, system-level privacy instructions, and skill-level intrusiveness labels. Results show that confidential-source access increases from 30% to 45% when users are unavailable, while urgency has little effect. Combining privacy instructions with intrusiveness labels reduces confidential-source access by roughly half compared to the baseline. The paper's main contribution is the benchmark and its empirical characterization of privacy-related source-selection behavior.

Critical strengths

First, the paper adopts a controlled experimental design by keeping the task-relevant information consistent across three acquisition pathways and counterbalancing tool presentation order. This helps reduce potential confounding factors when examining source-selection behavior. Second, the paper examines how different forms of privacy guidance affect source selection across five open-weight LLMs. The comparisons between system-level instructions, skill-level labels, and their combination provide some empirical evidence of how these factors influence

Critical weaknesses

First, the benchmark has limited task diversity and scope. Although it includes 55 tasks across 11 personal information categories, all tasks follow essentially the same three-source selection structure and require only a single missing attribute. This raises concerns about whether the benchmark captures sufficiently diverse privacy-related decisions and how well its findings generalize to more complex, real-world agent scenarios. Second, the benchmark's validity as a privacy evaluation is not sufficiently established. It relies on a fixed intrusiveness ordering among information sources and uses confidential-source access as its primary evaluation metric. Moreover, all sources are designed to provide identical task-relevant information, leaving out important factors such as source reliability, user burden, and contextual privacy requirements. These simplifications raise concerns about whether the benchmark can meaningfully evaluate privacy-sensitive source selection beyond its predefined setting.

Questions for authors
  1. The main text states that each model and experimental setting includes 3,300 valid runs, implying 99,000 valid runs across the main experiments. However, Appendix A reports only 81,377 total attempts, with an overall validity rate of 85.4%. Could the authors clarify this discrepancy and provide the actual number of valid runs used to compute the reported results?

  2. The benchmark assumes a fixed intrusiveness ordering among user interaction, public sources, and confidential sources. How is this ordering justified across different task contexts, particularly when user interaction involves photos, voice recordings, or signatures? To what extent can HIR serve as a reliable measure of privacy-sensitive source selection under this assumption?

  3. Given that all tasks follow the same single-attribute, three-source selection structure, could the authors provide more detailed evidence of how consistently the reported improvements hold across individual tasks and information categories? Are the observed effects broadly distributed or concentrated in a subset of scenarios?

1 comment

Thanks for your detailed review and for pointing out the inconsistency in the run counts. We have updated the attached manuscript accordingly. We address the concerns you raise and answer your questions below:

Critical Weaknesses.

Each task is deliberately constructed to involve 1 missing attribute and 3 potential information sources, each being sufficient to complete the task. This allows us to compare the choice of source, all other things being equal. The acquisition paths correspond to actions available to agents today. An agent can ask the user, read a public page, or access a private account (e.g., User interaction, LinkedIn Reader, Private Google Drive access). Our study reproduces these scenarios in a highly controlled setting. We leave tasks with multiple missing attributes or sources of varying reliability to future work.

We also detail the design of PrivacySkills in Appendix B.1 (lines 800–858). It aims to cover a broad range of privacy-related scenarios and agent decisions by drawing on four major privacy regulatory frameworks and cross-referencing the identified categories with the privacy practices of two major technology companies.

Q1: run counts. Thank you for catching this inconsistency. The main experiments contain 99,000 valid runs (5 models × 6 settings × 3,300). The values in the core of the paper are correct, but Tables 1 and 2 in Appendix A were produced from an intermediate snapshot of the logs. We apologize for this error. We have corrected the content in Appendix A (lines 757-787).

Q2: source ordering and HIR. When the user is available, all three skills provide the same task-relevant value. Only the confidential skill opens a restricted record and exposes additional personal information from it. As we note in the conclusion, asking the user for a photo or voice recording can make the assumed ordering inappropriate. 3 of our 55 tasks involve a photo, voice recording, or signature. In these cases, users may disclose more sensitive information than they would with either of the other two options. Although this limitation exists, we consider that HIR as defined in Section 3.6 is a reliable measure of privacy-sensitive source selection: (i) it measures the share of valid runs in which the agent accesses a restricted record even though a public source can provide the same value, and (ii) in the user unavailable condition, only the public and confidential sources can supply the attribute, so HIR directly measures how often the agent uses the confidential source when asking the user cannot resolve the task.

Q3: Variation across individual tasks and information categories. Thank you for this suggestion. We added Appendix B.2.2 (lines 900-966), which reports the relative HIR change per task (Fig. 7) and per category (Table 4) under each guidance condition. This shows that the results are consistent when labels are present (HIRs decrease for all categories, and almost all tasks, albeit with varying amplitudes), but not for instruction alone. The combined effect of label + instruction exceeds the sum of the separate effects in 10 of 11 categories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.