- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores how LLMs choose information sources that vary in sensitivity to complete tasks. To evaluate this question, an evaluation framework is developed by synthetically generating information sources as available skills, personal information categories, and tasks. The experimental results show that LLMs access confidential sources 30% on average, and including an intrusiveness metadata to the skills improve the intrusion rate.
Critical strengths
I think that the paper has some good strengths. The first main one is that the experimental design seems mostly solid. I like that the personal information categories are based on taxonomies from real-world legislation (GDPR, CCPA, LGPD, PIPL). Implementing the capabilities as skills following contemporary agent harnesses aligns well with the recent trend towards agentic frameworks. And controlling for any variation via deterministic outputs from the skills. The description of the experimental design is also well-detailed and documented, which can help with reproducibility.
Moreover, the paper addresses a timely problem of better understanding how LLMs make information-sharing decisions. The specific research questions explored are novel. To my knowledge, I don’t know of any works that have evaluated whether LLMs understand data source sensitivity.
Lastly, I like the results about the intrusiveness metadata as it well motivates the need for incorporating privacy annotations into skill specifications. I believe that this is a critical requirement for designing guardrails for agents, which nicely connects with the results from this paper.
Critical weaknesses
I have some reservations about the papers that lead me to weakly reject the work, but I would be open to changing my score if these reservations are clarified.
The first main reservation may be too open-ended to address, but the significance of the work feels a bit shaky to me. While I do agree that evaluating the problem of how LLMs choose data sources with varying sensitivity is interesting, I’m not entirely convinced that the problem and the insights derived from the results significantly contribute to the existing AI privacy work, at least to meet the ICLR significance bar. But I do think that the evaluation framework proposed in this work is a good contribution.
The second main reservation I have is the models evaluated in the work. Since they are all open-source models, I’m unsure if the results obtained from evaluations are limited only to open-source models. For example, it could be the case that frontier models rely on confidential sources significantly less than open-source models. In that case, the insights from the current results may not hold for more capable models. It would be extremely useful to include results from models such as the recent versions of GPT or Sonnet.
Lastly, I had a few minor critiques.
- For RQ 2, it would be more insightful to include the Acquisition tiers distribution per model. In particular, does the intrusiveness metadata just cause the LLMs to ask the user more?
- It would also be insightful to include an example containing the task, the specific user, public, and confidential source skills in the main text.
- There’s a lot of emphasis on valid runs through the main text. It would help to supplement Figure 3 by also including the number of total runs and valid runs together with each model to get a sense of the success rate in the context of source selection.
Questions for authors
- How many runs were attempted for each model in Figure 3? And how many of them were valid? Could you also explain to me the connection between Table 1 in Appendix A and the main results? Did each model get a different number of attempted runs?
- Did you do an error analysis for the sources of why certain runs would be invalid?
5 comments
Response by Reviewer
I thank the authors for their rebuttal. It has clarified most of the concerns that I had in my initial review.
However, I still have some concerns based on the results from the proprietary models. Seemingly, providing a simple instruction to "protect the user’s privacy and prefer the least intrusive source sufficient for the task" reduces the HIR to near zero for GPT-5, GPT-6 Astra, and Sonnet 5.5. Hence, this suggests that the benchmark might be too easy for frontier models, whereas providing this instruction for open-weight models had little effect. I understand that the simplified design of the benchmark allows for precise evaluation and attribution of information source sensitivity, but, as pointed out by reviewer XiVk, this simplification may be too unrealistic without enough complexity for frontier models.
Thanks for the follow-up discussion
We thank the reviewer for the thoughtful follow-up. We agree that a simple privacy instruction brings HIR close to zero for GPT-5, GPT-6 Astra, and Claude Sonnet 5.5 in our controlled setting. This is an encouraging result, but we believe it refines rather than undermines the findings of our study.
First, regarding RQ1 (propensity to access confidential sources): Without explicit privacy guidance, GPT-5, GPT-6 Astra, and Claude Sonnet 5.5 still access confidential sources in 27%, 12%, and 21% of valid runs, respectively, despite sufficient public alternatives. These rates increase to 37%, 27%, and 24% when the user is unavailable. Thus, even frontier models do not systematically avoid unnecessary confidential access without guidance. This also addresses the possibility raised in your initial review that frontier models might rely substantially less on confidential sources than open-weight models: our results show that this is not consistently the case.
Second, regarding RQ2 (impact of privacy guidance): The same instruction has little effect on GPT-5.4 mini (31% to 29% HIR), GPT-5.4 nano (22% to 19%), and the five open-weight models (30% to 29% on average). For these models, explicit source-intrusiveness labels provide additional improvements. Combining labels and instructions reduces HIR to 16% for mini, 3% for nano, and 15% on average for open-weight models. Our proprietary-model evaluation therefore provides a more nuanced conclusion: a simple instruction can be sufficient for certain frontier models in this setting, but its effectiveness is strongly model-dependent. This distinction is practically relevant, as smaller models also serve cost- and latency-sensitive deployments.
Finally, regarding task complexity: PrivacySkills is not primarily intended as a model-ranking challenge, but as a controlled framework for studying information acquisition decisions. By keeping the task-relevant information and available acquisition pathways fixed, we can isolate agents' propensity to access confidential sources and measure how this behavior changes with privacy guidance. This design remains informative even when some models achieve near-zero HIR under explicit instructions. We agree, however, that our findings do not establish whether these instructions remain sufficient in more complex, multi-step scenarios involving competing sources and contextual constraints. This is an important direction for future work, and we will make this limitation and the refined interpretation of RQ2 more explicit.
We thank the reviewer again for the constructive discussion and for acknowledging that most of the initial concerns have been clarified. We hope that the additional experiments on proprietary models, together with our clarification of the study's objectives and scope, address the remaining concerns about the significance of our findings.
We thank you for your careful reading of our paper and for your constructive feedback. We have also updated the attached manuscript. We address the concerns and questions raised below:
Critical Weaknesses: Significance. Source choice is important for the privacy protection later on, because it determines which personal data (e.g., a private medical document vs. a public profile) enters the context of the LLM, in order to extract a same missing attribute to resolve a task. Our experiments identify two factors that affect this choice. First, confidential accesses are performed even when a public source could be used and the user is available, and it increases when the user is unavailable. Second, a privacy instruction alone is not sufficient to remediate the problem but combining privacy instruction with skill labels provides better results. These findings foster incorporating privacy annotations into skill specifications and evaluating their effectiveness.
Critical Weaknesses: Proprietary models. Thank you for this suggestion. We added Appendix F (lines 1543-1619), which evaluates five proprietary LLMs on our benchmark: GPT-5, GPT-5.4 mini, GPT-5.4 nano, GPT-6 Astra, and Claude Sonnet 5.5. Due to cost and time constraints, these evaluations use 3 valid runs per task, setting, and tool permutation, rather than the 10 valid runs used for open-weight models. Our findings largely carry over:
Critical Weaknesses: Acquisition tiers distribution per model. We now present in Appendix G and in Fig. 11 (lines 1620-1652) how the acquisition tiers are distributed with and without guidance, for each model. On average, the privacy instruction alone barely reduces HIR (−1.1 pp, from 30.0% to 28.9%), labels alone yield a moderate reduction (−7.3 pp, i.e., −24% in relative terms), and combining both roughly halves it (−15.1 pp, to 15%), with the largest reduction for every model. This reduction does not merely shift runs toward asking the user: under label + instruction, the public source absorbs most of it (+12 pp on average), while user runs increase by only 3 pp.
Critical Weaknesses: Example. We have added a detailed example in Section 3.1 and Fig. 3 (lines 211-224), containing the task, the specific user, public, and confidential source skills. This makes explicit that all three pathways provide the same required value, while confidential access exposes additional personal information.
Critical Weaknesses: valid runs and Fig. 3. Thank you for this suggestion. We have supplemented Fig. 4 (the old Fig. 3) with the validity rate (lines 378-391). We have also extended Appendix A with a complete analysis, and we now provide the rate of valid and successful runs in our experiments for each model (lines 737-796).
Q1: attempts and Valid runs. For each (model, setting) pair, we run each of the 55 tasks under each of the 6 presentation orders for the skills until we obtain 10 valid runs. This gives 3,300 valid runs per cell. Consequently, models and settings with lower validity rates require more attempts (e.g., Mistral needs more attempts). This is now detailed in the complement provided in Fig. 4 (lines 378-391) and the extended Appendix A (lines 737-796). Table 1 pools all settings for each model. We found that the previous version was produced from an intermediate snapshot of the logs. We apologize for the inconsistency. The main results are unaffected by this reporting error. We have corrected Tables 1 and 2.
Q2: Error analysis. We also extend Tables 1 and 2 in Appendix A (lines 736–796) with a comprehensive error analysis by failure type:
The absence of acquisition accounts for the vast majority of invalid runs, representing 15.2 of the 17.2 pp of invalid runs.