- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLMs can recognize privacy risks when benign users unintentionally disclose sensitive information during ordinary interactions. The authors introduce PrivAwareBench, a benchmark designed to evaluate LLMs' ability to provide privacy risk warnings and suppress sensitive content under different settings, including single- and multi-exposure scenarios and varying context lengths. Experiments on 18 LLMs reveal that models do not consistently exhibit proactive privacy awareness in terms of warning and redaction behavior. The results also show that performance degrades when sensitive information appears in longer contexts.
Critical strengths
-
This paper addresses an underexplored privacy problem: benign users may unintentionally disclose sensitive information when interacting with LLMs. This is a practical and important problem in real-world scenarios, distinguishing the work from prior research that primarily focuses on malicious attacks or privacy leakage originating from models.
-
The authors introduce a new benchmark and evaluation protocol for this problem, including an assessment of whether models provide privacy risk warnings. The benchmark covers multiple types of sensitive information and various settings, such as repeated exposure and varying context lengths, enabling a broader evaluation of privacy-aware behavior.
Critical weaknesses
1.The scope of proactive privacy awareness is somewhat broader than what the evaluation actually measures. While the paper frames its contribution as evaluating proactive privacy awareness, the proposed evaluation primarily focuses on models' response-side behaviors. In practice, proactive privacy protection may also require system-level mechanisms that prevent sensitive information from being transmitted to LLMs or stored in logs. Therefore, I believe the scope of the paper's claims regarding proactive privacy awareness should be more clearly defined and limited to the behaviors being evaluated.
Furthermore, the paper seems to assume that privacy warnings and redaction should be aligned, but I am not fully convinced that issuing a warning is always necessary when sensitive content has already been suppressed. Redaction without an explicit warning may still provide sufficient privacy protection in certain scenarios. The authors should better justify why both behaviors are necessary and why their alignment is an important indicator of privacy awareness.
- Some experiments do not clearly distinguish privacy-specific failures from more general limitations of LLMs. For example, performance degradation in long-context settings is a well-known limitation of LLMs. Therefore, to support the claim that longer contexts introduce a particular challenge for privacy awareness, the authors should demonstrate whether privacy-related performance degrades more severely than general task performance under comparable conditions.
Similarly, in the reduced no-warning setting, a decrease in warning behavior could be interpreted as successful instruction following rather than a failure of privacy awareness. The authors should clarify what this setting is intended to measure and provide stronger justification for interpreting the observed behavior as a privacy-related failure.
Questions for authors
-
Could the authors provide further justification for why issuing privacy risk warnings is necessary for privacy-aware behavior, particularly when sensitive information is already suppressed through redaction?
-
Could the authors clarify the motivation and interpretation of the reduced no-warning setting? Additionally, could they compare performance degradation on privacy-related tasks with degradation on general tasks under long-context conditions to better isolate privacy-specific failures?
-
Could the authors conduct repeated sampling with the same prompts to assess per-instance variability in warning and redaction behavior? This would help determine whether the observed privacy-aware behaviors are consistent across multiple generations.
12 comments
What do the no-warning and input-length tests establish?
We have added task-completion and key-fact audits to the length experiment and incorporated the results into the revised paper. We address the two parts of your question separately: why the no-warning directive tests privacy robustness, and whether declining task completion explains the length trend.
Experiment 2 Privacy robustness under the no warning directive. The directive requests a direct answer, asks the model to omit warnings and masking advice, and claims that sensitive material has already been handled. The sensitive values remain in the input, and the claim supplies no evidence of revocation, a password change, or authorization for onward disclosure. The instruction therefore changes the requested response while leaving the exposure unresolved. Compliance that removes relevant protection is the robustness failure being tested. As with a request for bomb-making instructions, explicitly asking for an action does not establish that the model should follow it. Here the legitimate task can be completed with suppression and brief exposure-specific advice.
Table R3. Configuration-level changes under the combined no-warning instruction
| Category | Warning rate | Warning down / up / tie | Suppression rate | Suppression down / up / tie |
|---|---|---|---|---|
| Token | 51.87% → 35.14% | 85.71% / 14.29% / 0.00% | 55.63% → 58.86% | 28.57% / 71.43% / 0.00% |
| Password | 4.50% → 0.83% | 92.86% / 7.14% / 0.00% | 17.10% → 8.38% | 85.71% / 14.29% / 0.00% |
| Personal ID | 23.90% → 3.93% | 100.00% / 0.00% / 0.00% | 19.67% → 18.88% | 50.00% / 50.00% / 0.00% |
| Financial | 7.48% → 2.45% | 92.86% / 0.00% / 7.14% | 9.09% → 6.38% | 64.29% / 35.71% / 0.00% |
| Overall | 21.94% → 10.59% | 92.86% / 5.36% / 1.79% | 25.37% → 23.13% | 57.14% / 42.86% / 0.00% |
Warnings decrease in 92.86% of matched model-category configurations. Suppression decreases in 57.14% and increases in 42.86%; Token suppression rises on average. This separates reduced warning robustness from suppression behavior. The labels describe the observed response outcomes and do not identify an internal recognition mechanism.
Experiment 3 Task completion across lengths. GPT-5.6 evaluates each reply against the material actually visible in its input, under the same configurations, token positions, and nominal lengths. API errors and empty replies, at 0.08% and 0.05%, count as failures. R4a aligns task completion with the original end-position privacy series.
Table R4a. Privacy scores and task completion across nominal lengths at the end position
| Nominal length | WPR | Suppression | Task completion U |
|---|---|---|---|
| 256 | 69.07% | 32.43% | 72.29% |
| 512 | 59.43% | 33.29% | 72.21% |
| 1K | 29.50% | 33.21% | 69.64% |
| 1.5K | 20.79% | 58.64% | 63.57% |
| 2K | 15.64% | 66.36% | 58.57% |
Warning falls from 69.07% to 15.64%, compared with U from 72.29% to 58.57%. Suppression follows a different trend. R4b checks whether the warning-versus-delivery comparison depends on position or the length interval. Differences and rates are rounded separately.
Table R4b. Rate changes across nominal lengths (longer minus shorter, percentage points)
| Position and length | ΔWPR | ΔU |
|---|---|---|
| Front 256→2K | −45.43 | −13.43 |
| Middle 256→2K | −45.57 | −12.79 |
| End 256→2K | −53.43 | −13.71 |
| End 1K→2K | −13.86 | −11.07 |
Over 256-to-2K, warnings decline more than U at every position. Over end-position 1K-to-2K, the declines are closer, at 13.86 and 11.07 points. A matched end-position audit retains corresponding inputs with the same request ID and HTTP status: U decreases by 15.74 points, and a separate response-level warning rule decreases by 38.50 points. Requiring usable delivery and retention of both facts gives U2 of 57.87% and 52.42%. These subset warning labels are separate from the aggregate series. Code extent and padding vary across settings, so the results characterize these constructed variants.
The 1,500 variants share 100 base identifiers. The length labels are nominal. A post-hoc o200k_base audit of archived user text, excluding chat wrappers and pooling the three positions, gives median lengths of 140.5, 251.5, 402, 594.5, and 794.5 for the five settings. The appendix reports ranges and sensitive-span envelopes. Matching retains base ID and configuration; input-truncation flags were not logged.
Changes: Sections 5.4 and 5.5 clarify what the constructed length variants and combined no-warning directive test. Appendix G.4.1 and Table 12 add the task-completion and key-fact audits, reference-token length distributions, and sensitive-span measurements; Appendix E.4 clarifies context variation. Appendix G.6 and Table 13 report category-specific changes under the no-warning directive.
How consistent are repeated replies in each information category?
We have completed five calls per configuration on shared requests within each category and incorporated the results and consistency analysis into the revised paper. We report each model and category separately, then examine each request across all five calls to distinguish persistent success, persistent failure, and mixed outcomes.
Experiment 4 Repeated calls by category. Each configuration answers the same subset of 50 original requests per category. DeepSeek-V3.2 uses temperature 0.2 and top_p 0.95; the other configurations use endpoint defaults. Reissuing initially empty calls gives 100% scoring coverage, with source-text review of every recovered reply. GPT-5.6 evaluates U. A mixed W or R outcome means that a request-configuration group contains both successes and failures across the five calls.
R5a reports Token. These requests explicitly ask for log redaction or safe sharing, so the results measure repeat consistency under that stated instruction.
Table R5a. Token pass rates and within-request variation across five calls
| API configuration | WPR | R | Joint | U | W mixed rate | R mixed rate |
|---|---|---|---|---|---|---|
| gpt-5.6-sol | 97.2% | 77.6% | 76.4% | 100.0% | 8.0% | 36.0% |
| claude-opus-4-8 | 99.2% | 79.6% | 79.6% | 100.0% | 4.0% | 20.0% |
| gemini-3.8-flash | 58.0% | 36.0% | 20.8% | 100.0% | 26.0% | 14.0% |
| gpt-5.4 | 82.4% | 40.8% | 36.0% | 100.0% | 32.0% | 26.0% |
| gemini-3-pro-all | 68.0% | 32.0% | 22.8% | 100.0% | 40.0% | 28.0% |
| DeepSeek-V3.2 | 49.6% | 24.0% | 12.8% | 100.0% | 38.0% | 8.0% |
Opus 4.8 and GPT-5.6 Sol have high Token Joint rates, while suppression is mixed on 20.0% and 36.0% of requests. R5b checks whether warnings and suppression remain aligned on Password.
Table R5b. Password pass rates and within-request variation across five calls
| API configuration | WPR | R | Joint | U | W mixed rate | R mixed rate |
|---|---|---|---|---|---|---|
| gpt-5.6-sol | 97.2% | 87.2% | 86.0% | 100.0% | 8.0% | 30.0% |
| claude-opus-4-8 | 95.2% | 24.8% | 23.2% | 100.0% | 22.0% | 54.0% |
| gemini-3.8-flash | 31.2% | 20.0% | 4.8% | 100.0% | 58.0% | 26.0% |
| gpt-5.4 | 60.0% | 25.6% | 21.6% | 100.0% | 56.0% | 42.0% |
| gemini-3-pro-all | 47.2% | 18.4% | 12.0% | 100.0% | 62.0% | 40.0% |
| DeepSeek-V3.2 | 20.4% | 4.4% | 2.0% | 100.0% | 44.0% | 12.0% |
GPT-5.6 Sol passes Password Joint on every call for 68.00% of requests. Opus 4.8 warns frequently, but suppression is mixed on 54.0% of requests. R5c applies the same analysis to Personal ID.
Table R5c. Personal ID pass rates and within-request variation across five calls
| API configuration | WPR | R | Joint | U | W mixed rate | R mixed rate |
|---|---|---|---|---|---|---|
| gpt-5.6-sol | 50.4% | 56.8% | 39.2% | 98.0% | 42.0% | 14.0% |
| claude-opus-4-8 | 78.0% | 44.0% | 37.6% | 98.4% | 40.0% | 48.0% |
| gemini-3.8-flash | 48.8% | 38.0% | 32.0% | 98.0% | 6.0% | 0.0% |
| gpt-5.4 | 40.4% | 36.0% | 19.6% | 95.2% | 48.0% | 32.0% |
| gemini-3-pro-all | 56.4% | 25.2% | 22.0% | 98.4% | 30.0% | 40.0% |
| DeepSeek-V3.2 | 14.8% | 14.8% | 7.2% | 96.0% | 22.0% | 10.0% |
Gemini 3.8 Flash has 0.0% mixed suppression and returns identical text across calls on 88.00% of Personal ID requests despite distinct response IDs. Stability can therefore coexist with persistent privacy failures. R5d checks Financial alongside task completion.
Table R5d. Financial pass rates and within-request variation across five calls
| API configuration | WPR | R | Joint | U | W mixed rate | R mixed rate |
|---|---|---|---|---|---|---|
| gpt-5.6-sol | 45.6% | 57.2% | 36.4% | 100.0% | 50.0% | 24.0% |
| claude-opus-4-8 | 51.6% | 29.2% | 25.2% | 100.0% | 48.0% | 28.0% |
| gemini-3.8-flash | 5.6% | 6.8% | 2.0% | 100.0% | 12.0% | 16.0% |
| gpt-5.4 | 16.4% | 28.8% | 15.2% | 100.0% | 22.0% | 22.0% |
| gemini-3-pro-all | 8.8% | 4.8% | 2.8% | 100.0% | 20.0% | 10.0% |
| DeepSeek-V3.2 | 5.2% | 0.4% | 0.0% | 100.0% | 8.0% | 2.0% |
Financial U is 100.0% for every configuration. Joint is mixed on 42.00% of GPT-5.6 Sol requests and 34.00% of Opus 4.8 requests. Task delivery is therefore more consistent than the combined privacy outcome in these cases.
Changes: Appendix G.7 adds the repeated-call protocol and analysis. Table 10(a1-a2), Rep rows, reports privacy and task outcomes for each model and category; Table 10(b) reports within-request W/R mixed rates. Table 9(b) adds task/privacy intersections for repeated calls. Section 5.6 directs readers to the repeated-call experiment.
What does the benchmark measure, and why warn after suppression?
Thank you for your careful review and detailed questions about scope and reliability. Your distinction between observable response behavior and broader system protection is insightful and helps state the benchmark scope precisely. Following your requests, we have (1) clarified the response behaviors measured by the benchmark and their relationship to input handling and system-level protection; (2) explained how exposure-specific advice supports user action alongside provider responsibilities for storage, retention, and deletion; (3) added task-completion and key-fact audits to the length experiment and explained what the no-warning directive tests; and (4) completed repeated-call evaluation with model-by-category results, pooled performance, and consistency within each request. Revisions primarily addressing your comments are highlighted in green.
The paper measures how an assistant handles sensitive information supplied during a legitimate task. We have made the relationship between that response capability and broader system protection explicit. The added task and safeguard experiments show what is delivered to the user, while the discussion below explains what a warning can help the user do next.
Experiment 1 What the response metrics establish. W measures exposure-specific risk communication or protective advice, R measures suppression throughout the reply, and Joint measures their co-occurrence. U records usable task delivery. Input filtering, access control, retention, and deletion are provider and application responsibilities. The response labels assess advice and delivered content; user comprehension and subsequent remediation are distinct outcomes.
Suppression limits further propagation in a new deliverable. Exposed credentials can also require revocation or a password change at the issuing service. Okta documented session tokens in support attachments being used for session hijacking and remediation through token revocation. Removing a token from a draft therefore leaves a separate action at the issuing service; exposure-specific advice can identify that action. Okta investigation
For a date of birth or other information that cannot simply be rotated, actions concern access and retention: revoking a shared link, deleting a conversation, or deleting a separately saved file. ChatGPT schedules deleted chats for permanent removal within 30 days, with documented security, legal, and de-identification exceptions; separately saved files require separate deletion. Gemini Apps states that human-reviewed chats can remain for up to three years after activity deletion. These service-specific limits determine what deletion advice can accomplish. ChatGPT retention · Gemini Privacy Hub
A warning can identify exposure the user has not recognized and direct the user to the relevant control. Providers and applications must implement access, storage, and deletion protections for retained material. OpenAI, for example, documents abuse-monitoring retention and eligible data-retention controls for its API. The assistant contributes through advice and the delivered result; the provider remains responsible for those system controls. API data controls
Changes: Section 4 clarifies the measured response behaviors, the role of exposure-specific advice, and provider and application responsibilities for input handling, storage, retention, and deletion. Section 5.6, Table 5, and Figures 6-7 add task-delivery and safeguard results. Appendix F.3.1 explains the task and suppression checks; Appendices G.1.1-G.2 and Tables 9 and 11 provide the detailed experiments.