acceptodds
Under review as a conference paper at ICLR 2027

UAL Semantic Guard (USG): Efficiently detecting UAL prompt leakage attacks on LLMs

Abstract

Several black-box mechanisms protect large language models (LLMs) against prompt data leakage, an attack that forces a model to leak data derived from users’ private prompts. However, we found that they are extremely vulnerable to user attribute leakage (UAL) inference. Unlike other prompt data leakage attacks, UAL inference specifically targets user data that is not explicitly provided but can be inferred. Our evaluation shows that, against such attacks, the best overall protection mechanism, Meta PromptGuard, drops to only a 5% detection rate. To tackle this problem, we introduce UAL semantic guard (USG), a black-box prompt data leakage detector that also efficiently detects UAL inference. USG relies on a key idea: augmenting a small model’s context and providing attack and benign examples improve its ability to detect prompt data leakage attacks. We leverage this idea and augment a local model to act as an LLM-as-a-Judge that blocks prompt processing if it can lead to prompt data leakage. We evaluate our prototype against 3600 adversarial queries spanning nine UAL-Inference variants and 3200 benign queries covering several user tasks. We also evaluate against several categories of prompt data leakage. We run each evaluation on four open-weight LLMs: Llama 2, Llama 3.1, Mistral-Nemo, and Qwen. Our results show that (1) USG achieves 100% detection across all UAL attack variants, (2) while maintaining a 1.19% false-positive rate on benign queries, (3) outperforming the best state-of-the-art approaches, which only reach 55.2%, (4) with no major impact on prompt pro- cessing time and energy usage. Additionally, it maintains a 100% detection rate against other prompt data leakage attacks, demonstrating overall robustness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
UAL, Prompt leakage, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2ZDMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a black-box LLM-as-a-Judge safety mechanism against user attribute leakage (UAL) inference, called UAL semantic guard (USG). The USG efficiently detects UAL inference by extracting three markers from the input prompt, (1) intention of the inference operation, (2) privacy-sensitive attributes in the prompt, and (3) a person-level target. These three markers collectively ensured UAL identification even under undetectable malicious intent. Their results demonstrated a 100% detection across all UAL attack variants with 1.19% FPR (on benign queries).

Critical strengths
  1. The idea of using LLM-as-a-Judge for identifying UAL inference by extracting three markers in the input prompts, seems novel safety application usage, specific to UAL inference attacks.
Critical weaknesses
  1. Paper writing needs to improved. References are marked incorrectly in the content. They are in-line with textual content for all sections and subsections, making it hard to read. Also, please follow consistent terminology usage throughout the paper, e.g., UAL inference, and do not use different variants at different places. This could confuse some readers.
  2. Figures 1-3 needs to updated for better representation and making them understandable to readers. Please refer recent AI/ML conference published papers for improving these Figures.
  3. Results presented in App. C should have been discussed in Sec. 6 in the main pages. The results presented in Sec. 6 does not effectively support their hypothesis (only one row of UAL robustness).
  4. Evaluation baselines are weak as only one guardrails model is used. There are other guardrails like ShieldGemma, LlamaGuard (1-3) and Qwen3Guard that can be used as baselines for better comparison of USG.
  5. Paper lacks a discussion section for presenting clarifications to readers on any missing or excluded information that authors felt irrelevant to their scope.
Questions for authors

All concerns and questions are listed in the Weaknesses section. Please try to address the concerns highlighted.

Reviewer_kdViOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UAL Semantic Guard (USG), a pre-inference detector for prompt data leakage that focuses on user attribute leakage (UAL) inference, where an attacker asks an LLM to infer unstated personal attributes, such as age, location, or income, from a user's text. USG uses a small local LLM (Qwen2.5-1.5B-Instruct) as a judge, which receives a semantic security policy and six benign and malicious demonstrations, and blocks the query when its risk score exceeds a fixed threshold. Experiments on UAL attacks built from SynthPAI profiles and on benign tasks across four open-weight target models demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper identifies a practical gap: requests that infer unstated personal attributes look like normal analysis tasks, and PromptGuard 2 blocks only two of the nine UAL variants.
  2. The proposed method is simple and training-free, and it blocks attacks before target-model inference, which reduces energy and latency on attack traffic.
Critical weaknesses
  1. PromptGuard 2 is designed for prompt injection, so it may not be a strong baseline for profiling requests. The paper lacks comparison with LLM-based guard models that support privacy or custom categories, such as Llama Guard and WildGuard, and an ablation of the same Qwen2.5-1.5B judge without the demonstrations. These results would show whether the gain comes from the proposed design or from prompting an LLM judge with a privacy policy.

  2. The evaluation setting appears close to the judge policy. All nine attack prompts ask for almost the same attributes listed in the policy (Figure 4, Appendix A), most benign tasks (e.g., summary, sentiment, translation) are exempted in the policy, and there are no hard negatives, such as summarizing an attribute already stated in the text. Evaluating requests for other attributes and wordings, and benign requests that mention personal attributes, would make the 100% detection and 1.19% FPR more convincing.

Questions for authors
  1. Table 1 reports 5.0% for D1 on UAL-Inference, but Table 2 reports 46.2%, and the abstract (Lines 015–017) and conclusion (Lines 465–467) use different values. What is the reason for this difference?
  2. No table reports USG on non-UAL attacks (Lines 030–032, 387), and the per-attribute and task-level results (Lines 393–396, 402) are not in Appendix C. Could the paper provide them?
  3. Were D2 and D3 given the same UAL definition as USG? Why does PromptGuard 2 have a 50% FPR on simple benign tasks, and is this related to the French prompts (Line 615)?
  4. Since the judge does not see the target model (Lines 257–259), are the 3,600 instances 900 distinct judge decisions (9×1009 \times 100)? If so, the Wilson CI (Line 370) should be computed on 900.
  5. Lines 261–264 say USG blocks a query when the judge classifies it as UAL, but the JSON in Figure 4 has no classification field. Which output is used for blocking?

Minor:

  1. Table 1: CV, RH, PCL, and PDL are not defined.
Reviewer_LZvFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces UAL Semantic Guard (USG), a mechanism designed to prevent LLMs from inferring and revealing sensitive personal information about users.

The authors identify a weakness in existing LLM security defenses: they can detect traditional prompt injection and data leakage attacks but often fail to detect requests that infer private information indirectly.

USG addresses this by using a small, local LLM to analyze the semantic intent of a prompt before allowing the main LLM to process it.

Critical strengths

The paper presents a relatively comprehensive experimental evaluation across four different open-weight LLMs, multiple UAL-inference attack variants, and diverse benign tasks. The authors also compare USG against several existing defense mechanisms and evaluate multiple metrics, including detection effectiveness, false-positive rate, inference latency, and energy consumption. These experiments provide useful insights into the effectiveness and computational trade-offs of the proposed approach.

Critical weaknesses

The paper's proposed UAL-inference attack formulations closely resemble established adversarial prompting strategies, including role-playing, pretextual framing, and indirect requests, as illustrated by earlier jailbreak techniques such as the Grandma Exploit. While the targeted information differs, the paper does not convincingly establish that these formulations introduce a fundamentally new attack mechanism.

The proposed USG framework relies on an additional LLM (Qwen 2.5-1.5B-Instruct) to classify potentially malicious prompts using a predefined security policy and a few-shot prompting strategy. Although the authors emphasize that their approach is training-free, its effectiveness fundamentally depends on the judge model's ability to consistently understand and follow the provided instructions. This introduces another source of uncertainty rather than providing a deterministic security guarantee.

Besides, this paper contains numerous bad-formatted citations, references are inserted directly into sentences without appropriate punctuation, making the text unnecessarily difficult to follow.

Questions for authors

Could the authors evaluate whether the USG judge prompt can be directly ingested into the system prompts of the target LLMs, rather than relying on a separate LLM-as-a-Judge? This approach could potentially reduce computational overhead by eliminating the need for an additional inference pass, while still providing comparable privacy protection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.