- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper examines whether hidden-state prompt-injection probes stay reliable when the application interface changes and, more importantly, whether standard repair procedures truly restore the intended detection capability. Using two released TaskTracker probes on Phi-3-medium and Mistral-7B-Instruct-v0.2, the authors alter only the arrangement of the task, retrieved document, and conversation history while holding the underlying task, document, and injected instruction fixed. They categorize failures into three levels: threshold failure, score failure, and feature failure. Their experiments show that interface changes break the frozen threshold in 11 of 12 shifted settings, yet refitting a linear readout on labeled examples restores performance on the conventional injected-versus-clean evaluation in all 11 broken settings. The paper then argues that such apparent repair can be deceptive. Repaired probes may show high false-positive rates under harmless distribution shifts, such as added benign sentences, user-provided instructions, and how-to steps, and may substantially miss attacks from unseen attack sources. Adding specific benign and attack types to the repair data improves performance on those particular types, but the gain does not reliably transfer to new attack sources. The paper therefore concludes that a repaired probe should not be validated solely on the distribution used for repair, but should also be tested against unseen harmless text and attack sources.
Critical strengths
- The paper identifies an important and practically relevant evaluation problem. Prompt-injection probes may be deployed in applications whose prompt templates, message structure, or conversational context evolve over time. Showing that an interface-only change can substantially degrade a previously effective probe highlights a realistic deployment concern. The distinction between repairing benchmark performance and recovering the intended detection capability is particularly useful.
- The threshold/score/feature decomposition offers a clean framework for diagnosing probe failures. Separating threshold failure from score failure is conceptually simple but useful, distinguishing cases where recalibration suffices from those requiring a new readout or even new representations. Empirically, both occur under interface shift.
- The paper goes beyond standard attack-vs-clean evaluation and exposes meaningful hidden failure modes. The harmless-edit experiments are arguably the strongest part: added sentences, user-originated instructions, and how-to steps show that a detector can achieve excellent apparent injection detection while actually responding to properties not specific to malicious instructions. The finding that a refitted detector may essentially become a detector for the attack distributions in its training data is important.
- The experimental methodology is unusually careful. The paper separates exploratory and confirmatory experiments, locks predictions before several experiments, reports per-cell rather than only aggregate behavior, examines threshold selection with held-out examples, and reports negative findings. This substantially increases confidence that the reported phenomena are not merely artifacts of post-hoc threshold tuning.
Critical weaknesses
- The current evaluation provides limited evidence that the observed failure modes generalize to other prompt-injection probes. The core experiments consider only two released probes from TaskTracker. Although they are built on different base models, they share the same probe formulation. It therefore remains unclear whether the observed failures under interface shift and subsequent repair are specific to TaskTracker or also arise in prompt-injection probes with different hidden-state representations, readout mechanisms, or detection objectives. Evaluating at least one substantially different probe would strengthen the generality of the conclusions.
- The interface shifts considered are relatively limited compared with the diversity of interfaces in real-world agentic applications. The paper mainly considers separate-turn and scripted-history variants, which provide a well-controlled evaluation but cover only a small subset of realistic interface changes. Deployed agents may involve different chat templates, system/user/tool message arrangements, structured tool outputs, and multiple retrieved documents. It is therefore unclear whether the same threshold- and score-level failures persist across these more diverse interface changes.
- The analysis of generalization across attack sources remains limited. An important finding is that a probe refitted on particular attack sources may fail to generalize to a new source. However, the evaluated attack sources differ simultaneously in several dimensions, including linguistic style, attack objective, length, and construction process. Consequently, the experiments do not clearly identify which distributional factors drive the observed generalization failures. More controlled experiments that isolate these factors would provide a better understanding of what the repaired probes actually learn.
- The paper provides limited mechanistic insight into why the repaired probes fail to generalize. The experiments use hard negatives and unseen attack sources to show that refitted probes may rely on spurious or attack-specific features, but this conclusion is inferred primarily from their detection behavior. The paper does not directly analyze how the relevant hidden-state features change after refitting or whether they encode more fundamental properties such as instruction provenance, authority, or semantic role. Such analysis would help explain not only that the repair fails to generalize, but also why it fails and what the repaired probe actually learns.
Questions for authors
- Generality across probes: Do the observed interface-shift and repair failures also occur for prompt-injection probes beyond TaskTracker, particularly those using different representations or detection mechanisms?
- Realistic interface shifts: Have you evaluated more realistic interface variations, such as different chat templates, tool messages, structured tool outputs, or multiple retrieved documents?
- Attack-source generalization: Which properties of the attack sources (e.g., linguistic style, attack objective, or length) are primarily responsible for the observed cross-source generalization failures?
7 comments
Reply part 2
Weakness 1, continued. test at least one substantially different probe.
We ran one, PIShield [10]. It shares our how-to failure, and our refit on its hidden states has the added-sentence failure at its training threshold. The check's two protocols fixed the predictions before any score existed and declared it exploratory (Appendix H).
PIShield reads a hidden state of Llama-3.1-8B-Instruct given the document alone. It catches nearly all BIPIA attacks and fires on no clean document. Against our prediction, it misses 0.236 of human-written attacks. It fires on most documents with a how-to step, and no threshold keeps both misses and these false alarms at or below 0.10. It rarely fires on documents with an added sentence from another article.
Our refit on its states, with no interface change, passes the attack-versus-clean test. At its training threshold it fires on 0.404 and 0.816 of added sentences. At the held-out threshold, the refit fitted on BIPIA attacks misses human-written attacks and fires on most how-to steps (Appendix H).
On a whole prompt, which it is not built for, PIShield fires on every clean document. A second test, also fixed in advance, delivered the document alone in four wrappers. A tool message broke its threshold in sample 2, narrowly (misses 0.132 [0.095, 0.180]), and no other wrapper did. All four wrappers kept the how-to failure.
Weakness 2 and Q2. The two shifts cover few real agent interfaces, and what about other chat templates, tool messages, structured tool outputs and several retrieved documents.
Thanks for raising this question. The revision adds five layouts, a trained tool role, a tool header and real tool content (Section 4.6, Table 4, Appendix G), and Appendix I adds a system role and a second trained tool role.
The five layouts are JSON tool output in a user turn, XML tags, the document in an assistant turn, three documents, and twelve earlier exchanges. The probes subtract a reference, the hidden state of a prompt with no document. We build it three ways (an empty document slot, a slot saying "No document was found.", and the task alone) and count a failure only when all three agree.
At the frozen threshold, error exceeds 0.10 under all three reference prompts in 10 of 20 cells, and is at most 0.10 under all three in none. The other 10 cells depend on the reference, and we leave them out of the count. We read Mistral-7B v0.2 at 0.5, chosen after the toolkit's 0.9 failed on its own interface; at 0.9 the count is 13.
The score also fails in 6 cells: with an empty slot, no threshold on the frozen score has error at most 0.10 there (point estimates, Appendix G).
Reply part 3
Weakness 2, continued. whether the same threshold- and score-level failures persist across these interfaces.
Yes, and they persist in the form you singled out, a repair of benchmark performance that does not recover detection. On the five layouts the revision adds, as on the submitted ones, passing the attack-versus-clean test does not show that a refit works, as the next three paragraphs show. A pass shows only that a repair works on the kinds of text it was tested on. Based on this finding, we propose accepting a repaired probe only after counting its misses and false alarms at its deployed threshold on harmless edits, and on harmless kinds and attack sources that neither its fit nor its threshold used. Without a refit, the frozen threshold fails under all three reference prompts in 10 of 20 cells and passes under all three in none (the other 10 depend on the reference; Mistral-7B v0.2 read at 0.5, and at the toolkit's 0.9 the count is 13). In 6 cells, with an empty slot, no threshold on the frozen score keeps error at or below 0.10 (point estimates, Appendix G).
Refitted on each layout, the both-sources refit passes the attack-versus-clean test in all 20 cells (point estimates). Yet the combined refit, refitted the same way, misses 0.55 to 0.97 of third-source attacks in 8 of 10 layout and model pairs (Section 4.6). This confirmatory result holds. The other two pairs are Phi-3-medium with JSON tool output and with three documents.
The training threshold's failure persists too. A later test, locked with our own timestamp, read these 20 cells again. At their training thresholds, the readout refit, with no harmless edits in its fit, and the hard-negative refit, with them as negatives, keep added-sentence false alarms at or below 0.10 in only 1 and 4 of 20 cells. A linear readout with more dimensions than training examples can separate its training set [11], so training pushes harmless training scores down and a threshold just above them sits too low for new text. Based on this, we propose setting the threshold on held-out scores of clean documents and harmless edits, never on training scores. We chose this rule after seeing the counts on the first interfaces, and this locked test confirmed that it stops added-sentence false alarms. Thresholds set on held-out scores of the edits keep added-sentence false alarms at or below 0.10 in 19 of 20 cells each, though the readout refit then passes the four-rate test in 15, short of the 16 we predicted (Section 4.2, Appendix E). So the rule stops added-sentence false alarms; it does not fix how-to steps or new attack sources.
On JSON tool output the both-sources refit also fires on 0.230 (Phi-3-medium) and 0.680 (Mistral-7B) of how-to steps (Appendix G).
All these layouts keep SQuAD passages [12]. JSON escaping changes BIPIA's code blocks, and the three-document and history layouts add other SQuAD text. Each refit here is one fit.
Weakness 2 and Q2, tool messages. Deployed agents use tool messages and arrangements of system, user and tool messages.
We tested two trained tool roles, one in your arrangement. On Llama-3.1-8B-Instruct, a test locked before its runs, with our own timestamp, puts the task in the system role, then a user request and a retrieval call, then the document in the tool role. All five predictions held. The both-sources refit carried from the model's own layout breaks (error 0.95 and 0.93). Refitted there, it passes the attack-versus-clean test yet fires on 0.22 and 0.14 of how-to steps, and the combined refit misses 0.29 of third-source attacks (Appendix I).
The released probes' templates reject or drop a tool message (Supplement S54), so we also used Mistral-7B v0.3. We carried the both-sources refit, with its threshold, from three user-turn layouts to two tool layouts. Where a reply and the question follow the tool message, its error is 1.00 and 0.99. With the tool message last, its error in sample 2 is 0.17 [0.06, 0.30]; there it hides no how-to failure, but the combined refit, carried the same way, misses 0.77 of third-source attacks.
On v0.3 the role, the declared-tools block and, with the tool message last, the token we read change together, so this tests the whole tool set-up. v0.3 has no released probe, and the carried threshold rests on the empty-slot reference only (Table 4, Appendix G).
Reply part 1
Thank you for the review, and for naming the harmless edits as the strongest part.
Yes, for two of our failures. Attention Tracker [2], which scores how much selected attention heads attend to a fixed instruction, shares them: its carried threshold breaks on two of four other layouts, and it fires on how-to steps. It does catch the third attack source on its own layout.
Bracketed ranges in these replies give 99.4% intervals for the confirmatory statistics of our second protocol, hashed before its runs, and 95% intervals elsewhere (verdicts in Table 27); a bracketed single number points to the References block at the end of the comment.
With its released heads on Mistral-7B v0.3, the detector passed our check against its published result. On its own layout, at a threshold set on clean documents alone, the share of how-to steps (added WikiHow steps [3]) it fires on is 0.88 above its share of clean passages. There it misses no attack from InjecAgent [4], Section 4.4's third attack source.
Its own layout's threshold, set on labelled pairs and carried to four others, breaks after a separate turn and with the scripted history (error 1.00 in both samples). On the tool and JSON layouts every interval contains 0.10. With test labels, some threshold has error at most 0.06 on each of the four (Table 4, Appendix G).
We tested two, and neither removes the failure. With third-source strings in its fit, the linear combined refit catches nearly all held-out strings of that source (Figure 3a), so adding the source helped on that source where changing the readout did not. Based on this, we recommend testing any readout, linear or not, on an attack source that no fit or threshold used, as step 5 of our checklist says (Section 5).
The both-sources refit is a linear readout fitted on BIPIA [5] and human-written attacks [6], with harmless edits as negatives, and the combined refit adds how-to steps to its labels and threshold (Sections 4.3 and 4.4). An RBF support vector machine and an MLP, fitted on the combined refit's data with their own worst-kind thresholds (which minimise the worst error on any kind), miss 0.62 [0.52, 0.72] to 0.92 of third-source attacks, averaged over two shifted interfaces. This confirmatory result holds: these two nonlinear families, at this budget, do not rescue the refit (Section 4.6). Where one misses fewer attacks than the linear refit, it fires more on how-to steps, a threshold trade, not a rescue.
Three details set the scope of our Attention Tracker results, a carried threshold that breaks on two of four other layouts and firing on how-to steps (Appendix G). Its released threshold was set on short texts and fires on most clean passages of its own layout. We attribute this to passage length, not the interface, and the locked comparison already uses a threshold set on these passages. Its instruction sits in a system message on its own layout and in a user turn elsewhere. Reselected heads with a new threshold per layout pass the attack-versus-clean test in 7 of 8 cells; we read this repair on that test alone.
Yes, for new kinds of text. In an exploratory run, PIGuard [7], a text classifier, shows both failures: it misses a new attack source and fires on how-to steps. It reads only the document, so it cannot see the interface. So PIGuard answers the half of Q1 about new kinds of text. For the half about interfaces, Attention Tracker, which reads attention heads, has a carried threshold that breaks on two of four other layouts (Table 4, Appendix G).
At its shipped threshold, PIGuard misses 0.62 of third-source attacks and fires on 0.34 of how-to steps, so the same harmless kind breaks a different class of detector. On the two sources our refits were fitted on, it does well. It misses 0.060 of human-written attacks and none of 25 BIPIA strings, which bounds that miss rate at 0.113 (Section 4.4, Appendix G). A small test on BIPIA strings came before the protocol and informed these predictions, as the protocol's ledger records.
An earlier read of another released guardrail, in Supplement S47, failed its positive control, so it does not bear on interfaces. Section 2 cites one outside result [8]. There an attention detector falls to near chance on tool outputs from AgentDojo [9] in most model families, where data and format change together.