- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The research question of this paper is that, what are LLM safety judges actually detecting when they flag indirect prompt injection? The authors separate the malicious task from the injection-style announcement. They find that judges respond more strongly to the announcement than to the actual unauthorized task. In contrast, the models’ hidden states contain information about both the task and whether the user authorized it. The authors further show that targeted fine-tuning can make the judge’s answer recover this hidden information. The paper argues that current evaluations may overstate prompt-injection detection, and judges should be tested by varying the actual task.
Critical strengths
-
The core research question is simple but important. The authors investigate whether a safety judge is detecting the actual unauthorized task or just the surrounding warning-like language, which can be practically useful in many security scenarios.
-
I appreciate the efforts to rule out simple shortcuts. Concretely, the authors compare against word-, character-, style-, and length-based baselines, and these baselines cannot explain the authorization result. This strengthens the claim that the model is capturing the relation between the user request and the tool output.
Critical weaknesses
-
One of my major concerns is that the paper’s main criticism may be overly broad. It does show that announcement-like wording can inflate detection performance. But this does not necessarily mean that existing benchmarks broadly fail to measure prompt-injection detection. Some systems may rely on several signals at once, and the paper mainly isolates one important failure mode. It is unclear to me how the conclusion/finding generalizes to existing studies.
-
Relevant to the previous point, the study focuses on a relatively small set of attack tasks. The controlled dataset is based on 62 InjecAgent tasks. Even though there are many paired examples, the actual range of underlying attack goals appears to be limited. It is therefore unclear to me how the insight generalizes to broader injection attacks.
Questions for authors
-
The results from controlled examples built from 62 InjecAgent tasks are stronger, while transfer to AgentDojo is much weaker for authorization. How do the authors interpret this gap?
-
The paper argues that current benchmarks can overestimate detection because judges respond strongly to announcement-like wording. How broad is this effect?