acceptodds
Under review as a conference paper at ICLR 2027

The Announcement, Not the Attack: What LLM Safety Judges Answer and What Their Hidden States Hold

Abstract

LLM judges are widely used to catch indirect prompt injection, but the answer misses much of what their own hidden states record. To see what the answer and the hidden states each respond to, we separate the two parts of an injection. Besides the task that the user never asked for, an injected tool output usually carries an announcement telling the agent to ignore all previous instructions, and an attacker can simply leave the announcement out. Existing benchmarks mix the two, and their attack runs also differ from the benign ones in length and wording. We therefore rebuild the 62 InjecAgent tasks as pairs of tool outputs that differ in exactly one sentence, the task or a past-tense report of the same event, and add or remove the announcement on its own. We report three findings. First, judges answer the announcement more than the attack. Five open-weight judges separate the announcement from a neutral sentence at 0.967 AUROC or better but the task from its report only at 0.641 to 0.855. Even when the question asks only about actions the user did not request, adding an announcement to a report that carries no task raises the false-positive rate from at most 14.1% to 19.0–90.6%. Second, a linear probe on the hidden states reads the attack itself. Scored on held-out tasks, it separates the task from its report at 0.955 to 0.999 and just as well from a benign imperative of the same length. It also reads whether the user asked for the task, separating authorized from unauthorized tasks at 0.902 to 1.000 against 0.580 to 0.856 for the answer, on records where word, character, style and length classifiers score 0.500. Third, fine-tuning the five judges on announcement-free pairs closes the gap on the task, raising the answer's task detection to 0.987 or better. Judge evaluations should therefore vary the task while holding the announcement fixed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, LLM safety judges, LLM agents, probing hidden states, interpretability, evaluation methodology, false positives, agent security

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4nbsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The research question of this paper is that, what are LLM safety judges actually detecting when they flag indirect prompt injection? The authors separate the malicious task from the injection-style announcement. They find that judges respond more strongly to the announcement than to the actual unauthorized task. In contrast, the models’ hidden states contain information about both the task and whether the user authorized it. The authors further show that targeted fine-tuning can make the judge’s answer recover this hidden information. The paper argues that current evaluations may overstate prompt-injection detection, and judges should be tested by varying the actual task.

Critical strengths
  • The core research question is simple but important. The authors investigate whether a safety judge is detecting the actual unauthorized task or just the surrounding warning-like language, which can be practically useful in many security scenarios.

  • I appreciate the efforts to rule out simple shortcuts. Concretely, the authors compare against word-, character-, style-, and length-based baselines, and these baselines cannot explain the authorization result. This strengthens the claim that the model is capturing the relation between the user request and the tool output.

Critical weaknesses
  • One of my major concerns is that the paper’s main criticism may be overly broad. It does show that announcement-like wording can inflate detection performance. But this does not necessarily mean that existing benchmarks broadly fail to measure prompt-injection detection. Some systems may rely on several signals at once, and the paper mainly isolates one important failure mode. It is unclear to me how the conclusion/finding generalizes to existing studies.

  • Relevant to the previous point, the study focuses on a relatively small set of attack tasks. The controlled dataset is based on 62 InjecAgent tasks. Even though there are many paired examples, the actual range of underlying attack goals appears to be limited. It is therefore unclear to me how the insight generalizes to broader injection attacks.

Questions for authors
  • The results from controlled examples built from 62 InjecAgent tasks are stronger, while transfer to AgentDojo is much weaker for authorization. How do the authors interpret this gap?

  • The paper argues that current benchmarks can overestimate detection because judges respond strongly to announcement-like wording. How broad is this effect?

Reviewer_iVjrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLM safety judges detect unauthorized instructions or respond to accompanying authority announcements. It constructs paired evaluations from InjecAgent and compares model answers with linear probes of hidden states. The experiments examine task discrimination and user authorization, with supplementary analyses on AgentDojo. Fine-tuning on pairs without announcements substantially improves task discrimination.

Critical strengths

S1. Separating announcements, injected instructions, and user authorization offers a useful evaluation perspective. The paired construction helps identify what drives a judge’s predictions and provides more informative diagnostics than aggregate attack detection scores.

S2. The analysis includes meaningful controls, including evaluation on tasks excluded from training, balanced authorization records, alternative benign sentences, and matched agent records. These controls strengthen the reported observations, while the fine-tuning experiments demonstrate that task discrimination can be improved through appropriate supervision.

Critical weaknesses

W1. It remains unclear whether the main findings and fine-tuning gains hold for recent strong attacks that use fake conversations or mix malicious instructions with the user’s task. This limits the evidence that the proposed evaluation captures judge weaknesses across different attack types.

W2. Table 2 evaluates the larger API judges under the broad question, while Table 3 reports results under the task question only for the five open models. The paper therefore leaves unclear whether larger judges show the same false positives when asked specifically about actions the user did not request.

W3. The authorization probe does not transfer well to AgentDojo without retraining, as acknowledged in Section 5. This raises concerns about whether the probe can reliably identify user authorization in new agent environments.

Questions for authors

Q1. Does the gap between judge answers and probes remain for recent strong attacks, such as ChatInject[1]?

  • Results on these attacks would help clarify how widely the findings apply.

Q2. How do the larger API judges perform under the task question on E4 and E5?

  • Please report E4 AUROC and E5 false positive rates, with reasoning enabled where supported.

Q3. What explains the authorization probe’s performance drop when transferred to AgentDojo without retraining?

  • Please also test whether this drop occurs on comparable prompt injection cases from ASB[2] or AgentLab[3].

Additional suggestion. It would also be useful to check whether the fine-tuning gains carry over to recent strong attacks.


[1] Chang, Hwan, Yonghyun Jun, and Hwanhee Lee. "Chatinject: Abusing chat templates for prompt injection in llm agents." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Zhang, Hanrong, et al. "Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Jiang, Tanqiu, et al. "Agentlab: Benchmarking llm agents against long-horizon attacks." arXiv preprint arXiv:2602.16901 (2026).

Reviewer_hoqKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Injection judges are meant to notice when a tool output tries to make an agent do something the user never asked for. An injection usually carries two things: an announcement ("ignore all previous instructions") and the unrequested task itself. This paper rebuilds the InjecAgent tasks as minimal pairs in which each can be present or absent. The intuition is sharp: if a judge only reacts to the announcement, an attacker can simply leave it out. The answers of open judges track the announcement, flagging benign text that carries one and missing tasks that do not. A linear probe on the same forward pass separates tasks from past-tense reports and authorized from unauthorized requests, and fine-tuning on announcement-free pairs closes much of the gap.

Where it sits (October 2026): over-defence on trigger phrases is known (PIGuard and NotInject), the gap between what models recognise and what they do is concurrent work (Dong et al., 2026), and the field's current concern is whether high probe scores reflect shortcuts (arXiv 2606.22864 and arXiv 2602.14161, both cited). The new piece here is the factorisation of announcement and task, which is a genuinely useful test design.

Critical strengths
  1. The announcement test changes how judges should be evaluated. Holding the content fixed and adding only the announcement shows that open judges, trained classifiers and API judges all over-react to it, even when told not to count it. Any judge benchmark can adopt this test, and it directly informs defences: a judge that keys on announcements is trivially evaded.

  2. The minimal pairs control length and position, which most probing work on injections does not. The authors also show that length alone solves an existing benchmark, which makes the motivation concrete.

Critical weaknesses
  1. The probe claim is exposed to exactly the shortcut the 2026 literature warns about. The pairs contrast an instruction with a past-tense report of it, and in the released data every report contains a past-tense auxiliary while no task does, so a single word separates the classes and bag-of-words already matches the probe. The control meant to rule this out contrasts harmful tasks with benign ones, which tests harm rather than injection. Transfer is also weak: the probe degrades on AgentDojo, the authorization probe falls close to chance there, and adding the probe direction back moves the judge's answer no more than a random direction does. With a base of 62 tasks, "the hidden states read the attack" has not yet been separated from "the hidden states read the grammar of the construction".

  2. Part of the gap between answer and probe is a measurement choice. In minimal pairs the natural question is paired (does the judge score the task above its own report?), and on that metric the answer does well. The headline instead uses unpaired AUROC pooled over user contexts, a question that asks the judge to flag announcements, and text baselines that output a constant. The released code also cannot produce the clustered intervals that are reported.

Recommendation: lead with paired metrics and the task-only question, report fitted text baselines, and release the bootstrap actually used.

Questions for authors

Questions that could change my assessment:

  1. Does the probe still separate task from report when the reports are rewritten without past-tense auxiliaries or person markers, and what does a text baseline score on the benign-imperative control?
  2. How do answer and probe compare on paired win rates and under the task-only question, reported in the main tables?
  3. Which bootstrap produced the clustered intervals, and can you release it together with the data behind the main tables? Why does the retrained 27B adapter give a different E5 result from Table 6, and how many seeds stand behind that table?

Recommendations (not affecting my score):

  1. Compare with instruction detection from hidden states (arXiv 2505.06311) and with deployed guards (PromptGuard 2, LlamaFirewall) under the leave-one-dataset-out protocol of arXiv 2602.14161.
  2. Report per-fold means next to pooled out-of-fold AUROCs, since pooling across separately fitted folds shifts several numbers.
  3. Discuss SEP (Zverev et al., ICLR 2025) and the instruction hierarchy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.