- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a black-box LLM-as-a-Judge safety mechanism against user attribute leakage (UAL) inference, called UAL semantic guard (USG). The USG efficiently detects UAL inference by extracting three markers from the input prompt, (1) intention of the inference operation, (2) privacy-sensitive attributes in the prompt, and (3) a person-level target. These three markers collectively ensured UAL identification even under undetectable malicious intent. Their results demonstrated a 100% detection across all UAL attack variants with 1.19% FPR (on benign queries).
Critical strengths
- The idea of using LLM-as-a-Judge for identifying UAL inference by extracting three markers in the input prompts, seems novel safety application usage, specific to UAL inference attacks.
Critical weaknesses
- Paper writing needs to improved. References are marked incorrectly in the content. They are in-line with textual content for all sections and subsections, making it hard to read. Also, please follow consistent terminology usage throughout the paper, e.g., UAL inference, and do not use different variants at different places. This could confuse some readers.
- Figures 1-3 needs to updated for better representation and making them understandable to readers. Please refer recent AI/ML conference published papers for improving these Figures.
- Results presented in App. C should have been discussed in Sec. 6 in the main pages. The results presented in Sec. 6 does not effectively support their hypothesis (only one row of UAL robustness).
- Evaluation baselines are weak as only one guardrails model is used. There are other guardrails like ShieldGemma, LlamaGuard (1-3) and Qwen3Guard that can be used as baselines for better comparison of USG.
- Paper lacks a discussion section for presenting clarifications to readers on any missing or excluded information that authors felt irrelevant to their scope.
Questions for authors
All concerns and questions are listed in the Weaknesses section. Please try to address the concerns highlighted.