acceptodds
Under review as a conference paper at ICLR 2027

Do Not Assume Users Are Privacy-Aware: Evaluating Proactive Privacy Risk Warning in Large Language Models

Abstract

The widespread deployment of large language models (LLMs) has made safety a critical research priority. Existing studies predominantly focus on adversarial threats, such as harmful generation, and model-side privacy leakage. However, this assumes risk originates from malicious attackers, overlooking a common real-world scenario: benign users may unintentionally include sensitive information even when making legitimate requests. Such inclusion creates privacy risks that LLMs should proactively recognize, even without explicit privacy instructions. This calls for models to go beyond task completion by alerting users to these risks and preventing unintended data propagation. To evaluate this issue, we introduce PrivAwareBench, a benchmark designed to evaluate LLMs’ awareness of privacy risks arising from benign users’ unintentional disclosure. It assesses how this awareness is reflected in two observable behaviors: risk warning and content suppression, across diverse exposure conditions. The benchmark data are constructed with human involvement and manual quality control, and the evaluation judge is selected through a human–LLM agreement study. Experiments on 18 representative LLMs reveal that proactive privacy-aware behavior remains limited, category-sensitive, and brittle. These results expose a blind spot in protecting benign users from unintended disclosure.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Proactive Privacy Protection, Sensitive Information Exposure, Risk Warning, Content Suppression.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_p4QVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can recognize privacy risks when benign users unintentionally disclose sensitive information during ordinary interactions. The authors introduce PrivAwareBench, a benchmark designed to evaluate LLMs' ability to provide privacy risk warnings and suppress sensitive content under different settings, including single- and multi-exposure scenarios and varying context lengths. Experiments on 18 LLMs reveal that models do not consistently exhibit proactive privacy awareness in terms of warning and redaction behavior. The results also show that performance degrades when sensitive information appears in longer contexts.

Critical strengths
  1. This paper addresses an underexplored privacy problem: benign users may unintentionally disclose sensitive information when interacting with LLMs. This is a practical and important problem in real-world scenarios, distinguishing the work from prior research that primarily focuses on malicious attacks or privacy leakage originating from models.

  2. The authors introduce a new benchmark and evaluation protocol for this problem, including an assessment of whether models provide privacy risk warnings. The benchmark covers multiple types of sensitive information and various settings, such as repeated exposure and varying context lengths, enabling a broader evaluation of privacy-aware behavior.

Critical weaknesses

1.The scope of proactive privacy awareness is somewhat broader than what the evaluation actually measures. While the paper frames its contribution as evaluating proactive privacy awareness, the proposed evaluation primarily focuses on models' response-side behaviors. In practice, proactive privacy protection may also require system-level mechanisms that prevent sensitive information from being transmitted to LLMs or stored in logs. Therefore, I believe the scope of the paper's claims regarding proactive privacy awareness should be more clearly defined and limited to the behaviors being evaluated.

Furthermore, the paper seems to assume that privacy warnings and redaction should be aligned, but I am not fully convinced that issuing a warning is always necessary when sensitive content has already been suppressed. Redaction without an explicit warning may still provide sufficient privacy protection in certain scenarios. The authors should better justify why both behaviors are necessary and why their alignment is an important indicator of privacy awareness.

  1. Some experiments do not clearly distinguish privacy-specific failures from more general limitations of LLMs. For example, performance degradation in long-context settings is a well-known limitation of LLMs. Therefore, to support the claim that longer contexts introduce a particular challenge for privacy awareness, the authors should demonstrate whether privacy-related performance degrades more severely than general task performance under comparable conditions.

Similarly, in the reduced no-warning setting, a decrease in warning behavior could be interpreted as successful instruction following rather than a failure of privacy awareness. The authors should clarify what this setting is intended to measure and provide stronger justification for interpreting the observed behavior as a privacy-related failure.

Questions for authors
  1. Could the authors provide further justification for why issuing privacy risk warnings is necessary for privacy-aware behavior, particularly when sensitive information is already suppressed through redaction?

  2. Could the authors clarify the motivation and interpretation of the reduced no-warning setting? Additionally, could they compare performance degradation on privacy-related tasks with degradation on general tasks under long-context conditions to better isolate privacy-specific failures?

  3. Could the authors conduct repeated sampling with the same prompts to assess per-instance variability in warning and redaction behavior? This would help determine whether the observed privacy-aware behaviors are consistent across multiple generations.

12 comments
AuthorsOpenReview ↗

What does the benchmark measure, and why warn after suppression?

Thank you for your careful review and detailed questions about scope and reliability. Your distinction between observable response behavior and broader system protection is insightful and helps state the benchmark scope precisely. Following your requests, we have (1) clarified the response behaviors measured by the benchmark and their relationship to input handling and system-level protection; (2) explained how exposure-specific advice supports user action alongside provider responsibilities for storage, retention, and deletion; (3) added task-completion and key-fact audits to the length experiment and explained what the no-warning directive tests; and (4) completed repeated-call evaluation with model-by-category results, pooled performance, and consistency within each request. Revisions primarily addressing your comments are highlighted in green.

The paper measures how an assistant handles sensitive information supplied during a legitimate task. We have made the relationship between that response capability and broader system protection explicit. The added task and safeguard experiments show what is delivered to the user, while the discussion below explains what a warning can help the user do next.

Experiment 1 What the response metrics establish. W measures exposure-specific risk communication or protective advice, R measures suppression throughout the reply, and Joint measures their co-occurrence. U records usable task delivery. Input filtering, access control, retention, and deletion are provider and application responsibilities. The response labels assess advice and delivered content; user comprehension and subsequent remediation are distinct outcomes.

Suppression limits further propagation in a new deliverable. Exposed credentials can also require revocation or a password change at the issuing service. Okta documented session tokens in support attachments being used for session hijacking and remediation through token revocation. Removing a token from a draft therefore leaves a separate action at the issuing service; exposure-specific advice can identify that action. Okta investigation

For a date of birth or other information that cannot simply be rotated, actions concern access and retention: revoking a shared link, deleting a conversation, or deleting a separately saved file. ChatGPT schedules deleted chats for permanent removal within 30 days, with documented security, legal, and de-identification exceptions; separately saved files require separate deletion. Gemini Apps states that human-reviewed chats can remain for up to three years after activity deletion. These service-specific limits determine what deletion advice can accomplish. ChatGPT retention · Gemini Privacy Hub

A warning can identify exposure the user has not recognized and direct the user to the relevant control. Providers and applications must implement access, storage, and deletion protections for retained material. OpenAI, for example, documents abuse-monitoring retention and eligible data-retention controls for its API. The assistant contributes through advice and the delivered result; the provider remains responsible for those system controls. API data controls

Changes: Section 4 clarifies the measured response behaviors, the role of exposure-specific advice, and provider and application responsibilities for input handling, storage, retention, and deletion. Section 5.6, Table 5, and Figures 6-7 add task-delivery and safeguard results. Appendix F.3.1 explains the task and suppression checks; Appendices G.1.1-G.2 and Tables 9 and 11 provide the detailed experiments.

AuthorsOpenReview ↗

What do the no-warning and input-length tests establish?

We have added task-completion and key-fact audits to the length experiment and incorporated the results into the revised paper. We address the two parts of your question separately: why the no-warning directive tests privacy robustness, and whether declining task completion explains the length trend.

Experiment 2 Privacy robustness under the no warning directive. The directive requests a direct answer, asks the model to omit warnings and masking advice, and claims that sensitive material has already been handled. The sensitive values remain in the input, and the claim supplies no evidence of revocation, a password change, or authorization for onward disclosure. The instruction therefore changes the requested response while leaving the exposure unresolved. Compliance that removes relevant protection is the robustness failure being tested. As with a request for bomb-making instructions, explicitly asking for an action does not establish that the model should follow it. Here the legitimate task can be completed with suppression and brief exposure-specific advice.

Table R3. Configuration-level changes under the combined no-warning instruction

CategoryWarning rateWarning down / up / tieSuppression rateSuppression down / up / tie
Token51.87% → 35.14%85.71% / 14.29% / 0.00%55.63% → 58.86%28.57% / 71.43% / 0.00%
Password4.50% → 0.83%92.86% / 7.14% / 0.00%17.10% → 8.38%85.71% / 14.29% / 0.00%
Personal ID23.90% → 3.93%100.00% / 0.00% / 0.00%19.67% → 18.88%50.00% / 50.00% / 0.00%
Financial7.48% → 2.45%92.86% / 0.00% / 7.14%9.09% → 6.38%64.29% / 35.71% / 0.00%
Overall21.94% → 10.59%92.86% / 5.36% / 1.79%25.37% → 23.13%57.14% / 42.86% / 0.00%

Warnings decrease in 92.86% of matched model-category configurations. Suppression decreases in 57.14% and increases in 42.86%; Token suppression rises on average. This separates reduced warning robustness from suppression behavior. The labels describe the observed response outcomes and do not identify an internal recognition mechanism.

Experiment 3 Task completion across lengths. GPT-5.6 evaluates each reply against the material actually visible in its input, under the same configurations, token positions, and nominal lengths. API errors and empty replies, at 0.08% and 0.05%, count as failures. R4a aligns task completion with the original end-position privacy series.

Table R4a. Privacy scores and task completion across nominal lengths at the end position

Nominal lengthWPRSuppressionTask completion U
25669.07%32.43%72.29%
51259.43%33.29%72.21%
1K29.50%33.21%69.64%
1.5K20.79%58.64%63.57%
2K15.64%66.36%58.57%

Warning falls from 69.07% to 15.64%, compared with U from 72.29% to 58.57%. Suppression follows a different trend. R4b checks whether the warning-versus-delivery comparison depends on position or the length interval. Differences and rates are rounded separately.

Table R4b. Rate changes across nominal lengths (longer minus shorter, percentage points)

Position and lengthΔWPRΔU
Front 256→2K−45.43−13.43
Middle 256→2K−45.57−12.79
End 256→2K−53.43−13.71
End 1K→2K−13.86−11.07

Over 256-to-2K, warnings decline more than U at every position. Over end-position 1K-to-2K, the declines are closer, at 13.86 and 11.07 points. A matched end-position audit retains corresponding inputs with the same request ID and HTTP status: U decreases by 15.74 points, and a separate response-level warning rule decreases by 38.50 points. Requiring usable delivery and retention of both facts gives U2 of 57.87% and 52.42%. These subset warning labels are separate from the aggregate series. Code extent and padding vary across settings, so the results characterize these constructed variants.

The 1,500 variants share 100 base identifiers. The length labels are nominal. A post-hoc o200k_base audit of archived user text, excluding chat wrappers and pooling the three positions, gives median lengths of 140.5, 251.5, 402, 594.5, and 794.5 for the five settings. The appendix reports ranges and sensitive-span envelopes. Matching retains base ID and configuration; input-truncation flags were not logged.

Changes: Sections 5.4 and 5.5 clarify what the constructed length variants and combined no-warning directive test. Appendix G.4.1 and Table 12 add the task-completion and key-fact audits, reference-token length distributions, and sensitive-span measurements; Appendix E.4 clarifies context variation. Appendix G.6 and Table 13 report category-specific changes under the no-warning directive.

AuthorsOpenReview ↗

How consistent are repeated replies in each information category?

We have completed five calls per configuration on shared requests within each category and incorporated the results and consistency analysis into the revised paper. We report each model and category separately, then examine each request across all five calls to distinguish persistent success, persistent failure, and mixed outcomes.

Experiment 4 Repeated calls by category. Each configuration answers the same subset of 50 original requests per category. DeepSeek-V3.2 uses temperature 0.2 and top_p 0.95; the other configurations use endpoint defaults. Reissuing initially empty calls gives 100% scoring coverage, with source-text review of every recovered reply. GPT-5.6 evaluates U. A mixed W or R outcome means that a request-configuration group contains both successes and failures across the five calls.

R5a reports Token. These requests explicitly ask for log redaction or safe sharing, so the results measure repeat consistency under that stated instruction.

Table R5a. Token pass rates and within-request variation across five calls

API configurationWPRRJointUW mixed rateR mixed rate
gpt-5.6-sol97.2%77.6%76.4%100.0%8.0%36.0%
claude-opus-4-899.2%79.6%79.6%100.0%4.0%20.0%
gemini-3.8-flash58.0%36.0%20.8%100.0%26.0%14.0%
gpt-5.482.4%40.8%36.0%100.0%32.0%26.0%
gemini-3-pro-all68.0%32.0%22.8%100.0%40.0%28.0%
DeepSeek-V3.249.6%24.0%12.8%100.0%38.0%8.0%

Opus 4.8 and GPT-5.6 Sol have high Token Joint rates, while suppression is mixed on 20.0% and 36.0% of requests. R5b checks whether warnings and suppression remain aligned on Password.

Table R5b. Password pass rates and within-request variation across five calls

API configurationWPRRJointUW mixed rateR mixed rate
gpt-5.6-sol97.2%87.2%86.0%100.0%8.0%30.0%
claude-opus-4-895.2%24.8%23.2%100.0%22.0%54.0%
gemini-3.8-flash31.2%20.0%4.8%100.0%58.0%26.0%
gpt-5.460.0%25.6%21.6%100.0%56.0%42.0%
gemini-3-pro-all47.2%18.4%12.0%100.0%62.0%40.0%
DeepSeek-V3.220.4%4.4%2.0%100.0%44.0%12.0%

GPT-5.6 Sol passes Password Joint on every call for 68.00% of requests. Opus 4.8 warns frequently, but suppression is mixed on 54.0% of requests. R5c applies the same analysis to Personal ID.

Table R5c. Personal ID pass rates and within-request variation across five calls

API configurationWPRRJointUW mixed rateR mixed rate
gpt-5.6-sol50.4%56.8%39.2%98.0%42.0%14.0%
claude-opus-4-878.0%44.0%37.6%98.4%40.0%48.0%
gemini-3.8-flash48.8%38.0%32.0%98.0%6.0%0.0%
gpt-5.440.4%36.0%19.6%95.2%48.0%32.0%
gemini-3-pro-all56.4%25.2%22.0%98.4%30.0%40.0%
DeepSeek-V3.214.8%14.8%7.2%96.0%22.0%10.0%

Gemini 3.8 Flash has 0.0% mixed suppression and returns identical text across calls on 88.00% of Personal ID requests despite distinct response IDs. Stability can therefore coexist with persistent privacy failures. R5d checks Financial alongside task completion.

Table R5d. Financial pass rates and within-request variation across five calls

API configurationWPRRJointUW mixed rateR mixed rate
gpt-5.6-sol45.6%57.2%36.4%100.0%50.0%24.0%
claude-opus-4-851.6%29.2%25.2%100.0%48.0%28.0%
gemini-3.8-flash5.6%6.8%2.0%100.0%12.0%16.0%
gpt-5.416.4%28.8%15.2%100.0%22.0%22.0%
gemini-3-pro-all8.8%4.8%2.8%100.0%20.0%10.0%
DeepSeek-V3.25.2%0.4%0.0%100.0%8.0%2.0%

Financial U is 100.0% for every configuration. Joint is mixed on 42.00% of GPT-5.6 Sol requests and 34.00% of Opus 4.8 requests. Task delivery is therefore more consistent than the combined privacy outcome in these cases.

Changes: Appendix G.7 adds the repeated-call protocol and analysis. Table 10(a1-a2), Rep rows, reports privacy and task outcomes for each model and category; Table 10(b) reports within-request W/R mixed rates. Table 9(b) adds task/privacy intersections for repeated calls. Section 5.6 directs readers to the repeated-call experiment.

Reviewer_5dyCOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on evaluating LLMs’ awareness of privacy risks arising from benign users’ unintentional disclosure. This work proposes PrivAwareBench and assesses risk warning and content suppression behaviours.

Critical strengths
  1. This work proposes PrivAwareBench, which evaluates LLMs’ awareness of privacy risks arising from benign users’ unintentional disclosure. The authors conduct extensive experiments on 18 representative LLMs.
  2. The authors suggest that current LLMs function as helpful task executors rather than privacy-protective assistants.
Critical weaknesses
  1. Could the author provide evaluation results for more cutting-edge models? For example, GPT-5.6 and above, Opus 4.8 and above, and Gemini 3.8-Flash and above.
  2. There are only four categories of sensitive data; PrivAwareBench can be subjected to various privacy defense methods before undergoing evaluation.
  3. The dataset focuses on synthetic data; is there any real-world data available for testing?
Questions for authors

Please refer to the weaknesses detailed above.

6 comments
AuthorsOpenReview ↗

Do newer models close the privacy gap across categories?

Thank you for your careful and encouraging review, and for recognizing the value of our work. Your questions about model coverage, sensitive-field selection, and realistic tasks help connect the benchmark to its intended use. Following your requests, we have (1) added GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.8 Flash, reporting privacy outcomes and task completion for every model and category; (2) restated the existing legal and technical basis for the four categories and clarified its connection to field selection; (3) added regex, CRF, and GLiNER2-PII comparisons to examine category coverage, individual fields, and task preservation; and (4) evaluated tasks adapted from public WildChat-1M requests, retaining separate results for each model and category. Revisions primarily addressing your comments are highlighted in orange.

Following your request to expand model coverage, we have evaluated the three additional configurations on the complete Normal request set and incorporated the results into the revised paper. All three answer the same requests. Exposure-specific advice can receive warning credit; suppression requires checking the whole reply. GPT-5.6 evaluates U from the original request and reply. Scheduled calls remain in the denominator, including Opus 4.8 Financial API errors.

Experiment 1 Additional models by category. R6a starts with Token and keeps WPR, suppression, Joint, and U together for each model.

Table R6a. Normal Token

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol98.67%83.00%82.33%100.00%
claude-opus-4-897.00%80.00%78.33%100.00%
gemini-3.8-flash62.33%55.00%38.33%100.00%

Token has high Joint for GPT-5.6 Sol and Opus 4.8, with 100.00% U for all three configurations. R6b tests whether that credential protection extends to Password.

Table R6b. Normal Password

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol98.67%74.33%74.33%100.00%
claude-opus-4-891.67%27.00%25.67%100.00%
gemini-3.8-flash22.67%20.00%4.00%100.00%

On Password, Opus 4.8 warns at 91.67% but suppresses at 27.00%; 72.00% of its warning successes still reproduce a registered value. The next table checks non-credential identifying information.

Table R6c. Normal Personal ID

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol68.33%49.67%42.00%97.33%
claude-opus-4-870.67%45.00%35.00%99.00%
gemini-3.8-flash33.33%23.33%14.67%99.67%

Opus 4.8 has a higher Personal ID warning rate than GPT-5.6 Sol, but a lower Joint rate. Ranking by warnings alone therefore misses suppression differences. R6d completes the comparison on Financial.

Table R6d. Normal Financial

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol58.67%61.33%51.00%100.00%
claude-opus-4-853.67%24.67%22.00%99.33%
gemini-3.8-flash4.33%5.33%0.33%100.00%

GPT-5.6 Sol has the highest Joint among the three added configurations in every category. Gemini 3.8 Flash completes Financial tasks at 100.00% but warns at 4.33%, with 0.33% Joint. The additional models confirm that useful task delivery and combined privacy protection remain distinct outcomes.

Changes: Appendix G.1.2 adds the three model configurations and their evaluation. Table 10(a1-a2), N rows, reports warning, suppression, Joint, and task completion by model and category; Token and Password completion appears in the table note, and Personal ID and Financial completion in the U columns. Section 5.6 directs readers to the additional-model comparison.

AuthorsOpenReview ↗

Why these four categories, and how do safeguards cover them?

Thank you for examining the basis of the sensitive fields and asking how safeguards change the results. The paper provides the legal and technical rationale for the four categories. We explain it here to make the link between the cited protections, selected fields, and operational groups clear, then use the added safeguard experiments to examine category coverage and individual fields.

We selected fields using the legal protections and disclosure consequences discussed in the paper. GDPR, CCPA as amended by CPRA, the FTC Safeguards Rule, and PIPEDA inform the coverage of identifying information, nonpublic financial information, and account-access credentials. Personal ID and Financial organize the first two groups; Password and Token distinguish passwords from programmatic access credentials. RFC 6750 additionally provides the bearer-token security rationale. Each field is assessed in its task context.

Experiment 2 Category coverage and individual fields. R7a presents the input-and-reply regex/CRF comparison for your field-coverage question. The original request and reply stay fixed; both detectors merge input and reply detections before exact replacement, and an independent registry scores reproduction. These are the same controlled conditions reported in R1d.

Table R7a. Regex and CRF protection across Normal categories

CategoryOriginal replyRegex safeguardToken-level CRF
Token48.76%3.87%44.89%
Password80.66%2.67%0.25%
Personal ID77.86%73.74%77.49%
Financial90.89%37.54%71.82%
Overall75.17%31.80%51.40%

Personal ID remains weakly covered by these configurations. R7b restricts each row to replies whose requests contain the indicated field, showing how input matching relates to remaining reproduction.

Table R7b. Input detection and output reproduction by Personal ID field

FieldInput detector matchOriginal reproductionWith safeguard
Government ID6.60%72.69%72.40%
Email address100.00%81.47%0.00%
Phone number16.00%83.71%70.57%
Postal address0.00%69.90%69.90%
Date of birth0.00%51.36%51.36%

Email matching reaches 100.00% and its reproduction falls to 0.00%. Phone matching is 16.00%; postal addresses and dates of birth are not matched by these rules. These results locate the remaining disclosure and motivate testing semantic detection under the same replacement policy.

Experiment 3 Semantic coverage and task preservation. R7c fixes input-only scanning and reply replacement for regex, GLiNER2-PII, and their union. It isolates detector choice; R7a instead scans both input and reply.

Table R7c. Recent learned detection and combined input-derived protection

CategoryOriginal replyInput regexGLiNER2-PIICombined
Token48.76%4.94%19.45%2.60%
Password80.66%11.01%26.92%8.83%
Personal ID77.86%73.74%3.55%3.55%
Financial90.89%39.75%35.71%26.24%
Overall75.17%34.47%21.23%10.71%

GLiNER2-PII lowers Personal ID reproduction to 3.55%, while regex retains an advantage on Token and Password. Their union gives 10.71% overall reproduction. The corresponding task audit gives GLiNER2-PII 22.50% strict completion and 17.50% strict completion with suppression; the union gives 20.00% strict completion with suppression. Together, the analyses measure additional coverage and the effect of removing task-required information.

The paper also compares generation with and without added privacy instructions. Those interventions change what the assistant receives before generation; these safeguards process its returned content before display. Both are assessed through privacy and task outcomes. Health and precise location can extend the field scope using the same principle: preserve authorized task-required information and test whether irrelevant sensitive details are suppressed.

Changes: Section 5.1 clarifies how the legal and technical basis already presented in Section 3 and Appendix E.1 informs field selection. Section 5.6 and Figure 6 add the category-level detector comparison. Appendix G.2.1 and Table 11(a) report detector and scan-scope comparisons; Table 11(c) reports field-level coverage. Appendix G.2.2 and Table 11(b) report the corresponding task-preservation audit.

AuthorsOpenReview ↗

How do models perform on tasks adapted from real user requests?

Following your request to examine realistic tasks, we have added tasks adapted from public WildChat-1M requests and incorporated their construction and results into the revised paper. The original benchmark uses routine scenarios; this additional experiment tests privacy behavior on publicly recorded user task goals with registered synthetic sensitive values.

Experiment 4 Adapted real user tasks by category. Adaptation retains the original task goal and necessary context, removes identifying details, and inserts registered synthetic values. Each category accounts for 25% of requests, covering troubleshooting, job applications, payment follow-ups, and customer support. All six configurations answer the same requests under the warning and whole-response suppression criteria, with GPT-5.6 evaluating U. R8a begins with Token. WildChat

Table R8a. Adapted real-user tasks, Token

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol86.00%94.00%84.00%100.00%
claude-opus-4-890.00%72.00%64.00%100.00%
gemini-3.8-flash42.00%54.00%24.00%100.00%
gpt-5.462.00%72.00%48.00%100.00%
gemini-3-pro-all60.00%54.00%38.00%100.00%
DeepSeek-V3.28.00%42.00%2.00%100.00%

All configurations complete the Token tasks, while Joint ranges from 2.00% to 84.00%. R8b checks whether equal warning performance also implies equal password protection.

Table R8b. Adapted real-user tasks, Password

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol92.00%92.00%88.00%100.00%
claude-opus-4-892.00%36.00%34.00%100.00%
gemini-3.8-flash34.00%38.00%4.00%100.00%
gpt-5.444.00%50.00%22.00%100.00%
gemini-3-pro-all36.00%44.00%16.00%100.00%
DeepSeek-V3.212.00%30.00%4.00%100.00%

GPT-5.6 Sol and Opus 4.8 both warn at 92.00%, while Joint is 88.00% and 34.00%. Actual suppression determines this difference. R8c tests everyday identifying fields.

Table R8c. Adapted real-user tasks, Personal ID

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol16.00%70.00%12.00%100.00%
claude-opus-4-810.00%36.00%6.00%100.00%
gemini-3.8-flash0.00%30.00%0.00%100.00%
gpt-5.412.00%56.00%4.00%100.00%
gemini-3-pro-all4.00%30.00%2.00%100.00%
DeepSeek-V3.20.00%34.00%0.00%100.00%

Personal ID Joint is at most 12.00% despite complete task delivery. R8d extends the comparison to Financial tasks such as payment follow-ups.

Table R8d. Adapted real-user tasks, Financial

ModelWPRSuppressionJointTask completion U
gpt-5.6-sol32.00%54.00%30.00%100.00%
claude-opus-4-820.00%22.00%6.00%100.00%
gemini-3.8-flash10.00%28.00%8.00%100.00%
gpt-5.414.00%42.00%12.00%100.00%
gemini-3-pro-all12.00%38.00%12.00%100.00%
DeepSeek-V3.26.00%22.00%4.00%100.00%

U is 100.00% in every model-category cell. Financial Joint ranges from 4.00% to 30.00%, and the differences across all four categories depend on both model and information type. The added experiment shows that privacy gaps persist on actual user task goals alongside useful delivery, supporting separate measurement of task completion, warnings, suppression, and their co-occurrence.

Changes: Appendix G.1.2 adds the WildChat-based task adaptation, source reference, and evaluation. Table 10(a1-a2), A rows, reports privacy and task outcomes for every model and category; its note records Token and Password completion, and its U columns report Personal ID and Financial completion. Section 5.6 directs readers to these adapted-task results.

Reviewer_4eqLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrivAwareBench to evaluate how LLMs respond when benign users include sensitive information in legitimate requests. The benchmark contains 3,100 synthetic instances covering tokens, passwords, personal identifiers, and financial information, organized into standard exposure, multi-category exposure, and context-length/position variants. It separately measures explicit privacy warnings, suppression of sensitive values, and their conjunction under proactive and assisted prompting. The authors evaluate 18 model configurations using an LLM judge validated against human annotations. Results show substantial variation across models and sensitive categories, frequent separation between warning and suppression, and improvements with assisted prompting. Warning behavior is sensitive to input length, sensitive-value position, and instructions discouraging warnings, while suppression exhibits less consistent patterns.

Critical strengths

1.The paper provides a systematic empirical characterization of how models handle unnecessary sensitive information in their outputs. These observations provide useful evidence for designing and evaluating safeguards.

Critical weaknesses
  1. The need for spontaneous model-level privacy intervention is not established. The paper assumes that models should spontaneously warn about and suppress sensitive information, but does not establish why this behavior must originate from the model itself. Input preprocessing can detect and remove sensitive values before transmission. A harness can also inspect both the input and the model’s response to decide whether to warn the user, suppress sensitive content, or request clarification. This separates privacy decisions from task generation and allows policies to reflect the application, recipient, and user preferences. In some authorized workflows, warnings or suppression may be unnecessary or undesirable. The authors should identify where spontaneous model behavior adds value beyond such safeguards and compare its effectiveness, consistency, and flexibility against harness-based alternatives. Without this justification, low proactive warning rates do not establish a practically important model deficiency.

  2. The evaluation protocol is too simplified to establish effective privacy-aware assistance. The benchmark evaluates only the first response through warning and suppression labels, without assessing whether the model completes the user’s task. Under the formal metric definitions, a response that warns and withholds assistance can satisfy the joint criterion simply by omitting sensitive values. The protocol also excludes interactions in which the assistant first warns or requests clarification and then answers after user feedback. It therefore cannot distinguish a productive privacy-aware interaction from a warning that leaves the task unresolved, or establish whether protection persists through eventual task completion. Evaluating task success and sensitive-information propagation across complete interactions would better substantiate the practical significance of the benchmark scores.

Questions for authors
  1. In which concrete deployment scenarios does spontaneous model-level warning and suppression provide a necessary or meaningful benefit beyond a harness that preprocesses inputs and inspects both inputs and outputs to decide whether to warn, suppress, or seek clarification?
  2. How does spontaneous model behavior compare with a simple harness-based approach that inspects the input and generated response and applies privacy interventions when needed? A comparison using the same underlying model, measuring both warning/suppression and task completion, would help establish the practical value of the capability being benchmarked.
  3. How does the evaluator score a response that warns the user and requests sanitized input without completing the task? Section 4 accepts omission as suppression, whereas Appendix J requires explicit redaction evidence. Would such a response receive full Joint credit, and how frequently does this behavior occur in the reported experiments?
7 comments
AuthorsOpenReview ↗

Why should task models handle privacy when external safeguards exist?

Response to W1 & Q1: Why should task models handle privacy when external safeguards exist?

Thank you for your careful reading and detailed suggestions. Following your requests, we have (1) explained the value of privacy capability within the task model through resource constraints, complementary protection on detector misses, and gaps in API-delivered replies; (2) added controlled comparisons with regex, CRF, and GLiNER2-PII safeguards, including privacy outcomes, strict task preservation, and deployment-resource measurements; (3) added GPT-5.6 evaluation of task completion across the complete Normal audit archive and reported its intersections with privacy outcomes; and (4) clarified the privacy-specific purpose of Joint and how omission provides whole-response suppression evidence. The detailed results distinguish models, categories, and evaluation protocols. Revisions primarily addressing your comments are highlighted in blue.

We address your question from three angles. Resource measurements establish the cost of a separate semantic detector; controlled comparisons test complementary protection; API response results show which privacy behaviors remain absent during otherwise successful task delivery.

Point 1 Privacy capability under resource constraints. An offline terminal or edge device budgeted for one small open-weight task model needs privacy handling within task generation when it cannot accommodate a separate semantic detector or call a cloud safeguard. Our GLiNER2-PII configuration uses 1.14 GiB of FP32 parameter data and reaches 3.23 GiB peak process RSS in a separate loading-and-inference probe, including runtime overhead. These measurements identify an additional resource requirement. In a device whose budget is occupied by the task model, exposure detection, relevant advice, and suppression need to occur within that available model.

Point 2 Complementary protection on detector misses. Where external safeguards can be deployed, the task model contributes when detection misses a field. In the audited subset where the regex input scanner missed every registered value, 29.76% of original replies supplied valid warnings and 9.52% combined warning, suppression, and strict task completion without a system notice. In the reverse direction, adding verified credential notices to regex protection raised warning coverage from 32.50% to 56.67% and warning, suppression, and strict completion from 7.50% to 14.58%. Each component supplies protection omitted by the other in this pipeline.

Point 3 Privacy gaps in API delivered replies. Providers document request classification, output filtering, and guardrail integrations with service-specific applicability. We evaluate API-delivered replies by checking exposure-specific warnings and suppression alongside task completion.

Against this documented service context, Gemini 3.8 Flash achieves 100.00% task completion on Financial with a warning rate of 4.33%. Opus 4.8 warns on 91.67% of Password tasks but suppresses values on 27.00%; 72.00% of its warning successes still reproduce a registered value. These delivered outcomes establish a continuing need to evaluate warning and suppression alongside task completion. Together, the resource evidence, detector-miss results, and API outcomes support privacy capability within the task model and complementary safeguards suited to the application.

Changes: Section 4 clarifies the response-level privacy capability and its complementary role with external safeguards. Section 5.6, Table 5, and Figures 6-7 add task-delivery and safeguard results. Appendix G.2.2 adds the detector-miss and credential-notice audits; G.2.3 reports detector resources and provider-documented safeguards. Appendix G.1.2 and Table 10(a1-a2), N rows, report the additional-model results by category.

AuthorsOpenReview ↗

Which safeguards do the tested API services document?

Response to W1 & Q1: Which safeguards do the tested API services document?

Providers document safeguards at different levels of their services. R1a distinguishes request classification and output filtering from separately invoked moderation services and guardrail integrations, and records their stated applicability.

Table R1a. Provider-documented safeguards and their applicability

Provider or serviceDocumented safeguardApplicability
OpenAIRequest-risk classification for GPT-5 and a separate Moderation API.Classifier checks and optional moderation are distinct service features. Safety classifiers · Moderation API
Anthropic Claude APIStreaming classifiers can interrupt a response with a refusal stop reason.The documented behavior concerns streaming API responses. Streaming refusals
Google Cloud Gemini and Gemini APIGoogle Cloud documents fixed sensitive-PII output filtering. The Gemini API documents adjustable harm-category filters.The two documents describe their respective services and settings. Cloud filtering · Gemini API settings
DeepSeek APIResponses can report content_filter when filtering omits content.The finish reason records a filtering event. API response specification
xAI APIRequest guidelines can add decline behavior for specified categories.Unspecified categories retain default moderation behavior. Moderation controls
Alibaba Cloud Model StudioAn AI guardrail can scan model inputs and outputs.The documented integration requires activation, authorization, and a request header, and is limited to the China (Beijing) region. Guardrail configuration

The documented features establish the service context for the API experiments. Warning, suppression, and task-completion scores evaluate the replies the user receives. This connects provider disclosures to the behavioral gaps in the model-by-category results.

Changes: Appendix G.2.3 adds provider-documented safeguards and their service-specific applicability. Appendix G.1.2 and Table 10(a1-a2), N rows, report the additional-model outcomes by category. Section 4 clarifies the scope of evaluating delivered responses in this service context.

AuthorsOpenReview ↗

How do external detectors change disclosure and task completion?

Response to W1 & Q2: How do external detectors change disclosure and task completion?

Following your request, we have added controlled safeguard comparisons and a strict task audit to the revised paper. Each condition starts from the same original request and reply. We first measure whether detection and replacement reduce disclosure, then check whether the protected reply still contains the required task result. Scan-scope and credential-notice comparisons identify what the external layer contributes; the detector-miss audit identifies what generation contributes. Resource measurements then establish the extra deployment budget. Together, these experiments test the value of safeguards and explain when privacy handling within the task model remains necessary.

Experiment 1 Input detection and reply replacement. The detector reads the original request; every exact occurrence of its detected values in the original reply is replaced with [PROTECTED_VALUE] before display. We compare regex, the public pretrained GLiNER2-PII checkpoint at threshold 0.5 with fixed category-specific labels, and their union. All Normal replies are processed. Reproduction is scored on the 89.17% with independently registrable values, using a scoring registry separate from detector inputs. GLiNER2-PII report · Checkpoint

Table R1b. Registered-value reproduction under a fixed input-derived redaction policy

CategoryOriginal replyInput regexGLiNER2-PIICombined
Token48.76%4.94%19.45%2.60%
Password80.66%11.01%26.92%8.83%
Personal ID77.86%73.74%3.55%3.55%
Financial90.89%39.75%35.71%26.24%
Overall75.17%34.47%21.23%10.71%

Overall reproduction falls to 21.23% with GLiNER2-PII and 10.71% with the union. GLiNER2-PII adds Personal ID coverage, while regex has lower reproduction on Token and Password. The complementary coverage explains the union improvement. We next check whether broader replacement preserves the information required for the task.

Experiment 2 Strict task preservation. R1c uses a common-request audit panel of GPT-5.4, Claude 4.5, and Gemini 3 Flash replies. Strict completion requires the main deliverable, essential facts, and constraints. Detector-only conditions supply replacement, so their warning labels remain the original model warnings. This separates detector-driven suppression from the advice generated by the task model.

Table R1c. Privacy and strict task completion under the controlled detector comparison

OutcomeOriginal replyInput regexGLiNER2-PIICombined
Original model warning32.50%32.50%32.50%32.50%
Suppression of all audited values39.58%72.08%82.08%90.83%
Strict task completion31.67%31.67%22.50%22.50%
Strict completion with suppression10.83%22.08%17.50%20.00%
Warning with suppression18.33%28.33%29.58%30.42%
Warning, suppression, and strict completion7.50%11.25%10.42%10.83%

Among the input-only conditions in R1c, the union achieves the highest suppression, 90.83%, while regex achieves the highest strict completion with suppression, 22.08%. GLiNER2-PII replacements sometimes remove required facts, requested identifiers, or code identifiers, reducing strict completion to 22.50%. Evaluating both outcomes identifies the privacy protection and task preservation delivered by each condition.

Changes: Section 5.6, Table 5(b), and Figure 6 add the controlled detector comparison and its task-preservation results. Appendix G.2.1 and Table 11(a) specify the detectors, scan sources, replacement policy, and reproduction scores. Appendix G.2.2 and Table 11(b) report warning, suppression, and strict task completion on the common audit panel.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.