acceptodds
Under review as a conference paper at ICLR 2027

Is a Repaired Prompt-Injection Probe Really Repaired?

Abstract

A prompt-injection probe reads a language model's hidden states and flags a retrieved document that carries an instruction the user did not give. When an application changes the interface, the way the task and the document are laid out in the prompt, such a probe can break. The usual fix is a new threshold or a new linear readout, judged by passing the test on attacks against clean documents again. In this paper, we first show that changing only the interface breaks two released TaskTracker probes in 11 of 12 cells, and that a new readout passes that test again in all 11. Then, we show that passing again can hide two failures that also occur without the shift: a threshold that fires on harmless edits of the document, and a score that misses attacks from another source. We find that adding a kind of text to the repair's data removes its failure on held-out text of that kind: a refit fitted on two attack sources, with harmless how-to steps in both its labels and its threshold, set per kind of text, passes all these tests in 78 of 80 random splits of its data (20 per shifted probe-interface pair). But on a new attack source it fails again: it misses at least 0.57 of its attacks, while the same refit without how-to steps catches them but fires on most how-to steps and quoted injections. So a repaired probe should be tested on harmless text and attacks that were not in its data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt injection detection, activation-based monitoring, LLM agent safety, interface shift, distribution shift, threshold calibration, probe transferability.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fzfCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether hidden-state prompt-injection probes stay reliable when the application interface changes and, more importantly, whether standard repair procedures truly restore the intended detection capability. Using two released TaskTracker probes on Phi-3-medium and Mistral-7B-Instruct-v0.2, the authors alter only the arrangement of the task, retrieved document, and conversation history while holding the underlying task, document, and injected instruction fixed. They categorize failures into three levels: threshold failure, score failure, and feature failure. Their experiments show that interface changes break the frozen threshold in 11 of 12 shifted settings, yet refitting a linear readout on labeled examples restores performance on the conventional injected-versus-clean evaluation in all 11 broken settings. The paper then argues that such apparent repair can be deceptive. Repaired probes may show high false-positive rates under harmless distribution shifts, such as added benign sentences, user-provided instructions, and how-to steps, and may substantially miss attacks from unseen attack sources. Adding specific benign and attack types to the repair data improves performance on those particular types, but the gain does not reliably transfer to new attack sources. The paper therefore concludes that a repaired probe should not be validated solely on the distribution used for repair, but should also be tested against unseen harmless text and attack sources.

Critical strengths
  1. The paper identifies an important and practically relevant evaluation problem. Prompt-injection probes may be deployed in applications whose prompt templates, message structure, or conversational context evolve over time. Showing that an interface-only change can substantially degrade a previously effective probe highlights a realistic deployment concern. The distinction between repairing benchmark performance and recovering the intended detection capability is particularly useful.
  2. The threshold/score/feature decomposition offers a clean framework for diagnosing probe failures. Separating threshold failure from score failure is conceptually simple but useful, distinguishing cases where recalibration suffices from those requiring a new readout or even new representations. Empirically, both occur under interface shift.
  3. The paper goes beyond standard attack-vs-clean evaluation and exposes meaningful hidden failure modes. The harmless-edit experiments are arguably the strongest part: added sentences, user-originated instructions, and how-to steps show that a detector can achieve excellent apparent injection detection while actually responding to properties not specific to malicious instructions. The finding that a refitted detector may essentially become a detector for the attack distributions in its training data is important.
  4. The experimental methodology is unusually careful. The paper separates exploratory and confirmatory experiments, locks predictions before several experiments, reports per-cell rather than only aggregate behavior, examines threshold selection with held-out examples, and reports negative findings. This substantially increases confidence that the reported phenomena are not merely artifacts of post-hoc threshold tuning.
Critical weaknesses
  1. The current evaluation provides limited evidence that the observed failure modes generalize to other prompt-injection probes. The core experiments consider only two released probes from TaskTracker. Although they are built on different base models, they share the same probe formulation. It therefore remains unclear whether the observed failures under interface shift and subsequent repair are specific to TaskTracker or also arise in prompt-injection probes with different hidden-state representations, readout mechanisms, or detection objectives. Evaluating at least one substantially different probe would strengthen the generality of the conclusions.
  2. The interface shifts considered are relatively limited compared with the diversity of interfaces in real-world agentic applications. The paper mainly considers separate-turn and scripted-history variants, which provide a well-controlled evaluation but cover only a small subset of realistic interface changes. Deployed agents may involve different chat templates, system/user/tool message arrangements, structured tool outputs, and multiple retrieved documents. It is therefore unclear whether the same threshold- and score-level failures persist across these more diverse interface changes.
  3. The analysis of generalization across attack sources remains limited. An important finding is that a probe refitted on particular attack sources may fail to generalize to a new source. However, the evaluated attack sources differ simultaneously in several dimensions, including linguistic style, attack objective, length, and construction process. Consequently, the experiments do not clearly identify which distributional factors drive the observed generalization failures. More controlled experiments that isolate these factors would provide a better understanding of what the repaired probes actually learn.
  4. The paper provides limited mechanistic insight into why the repaired probes fail to generalize. The experiments use hard negatives and unseen attack sources to show that refitted probes may rely on spurious or attack-specific features, but this conclusion is inferred primarily from their detection behavior. The paper does not directly analyze how the relevant hidden-state features change after refitting or whether they encode more fundamental properties such as instruction provenance, authority, or semantic role. Such analysis would help explain not only that the repair fails to generalize, but also why it fails and what the repaired probe actually learns.
Questions for authors
  1. Generality across probes: Do the observed interface-shift and repair failures also occur for prompt-injection probes beyond TaskTracker, particularly those using different representations or detection mechanisms?
  2. Realistic interface shifts: Have you evaluated more realistic interface variations, such as different chat templates, tool messages, structured tool outputs, or multiple retrieved documents?
  3. Attack-source generalization: Which properties of the attack sources (e.g., linguistic style, attack objective, or length) are primarily responsible for the observed cross-source generalization failures?
7 comments
AuthorsOpenReview ↗

Reply part 1

Weakness 1 and Q1. Both probes come from TaskTracker [1] and share one formulation. Whether the interface-shift and repair failures also occur in probes with other representations or detection mechanisms.

Thank you for the review, and for naming the harmless edits as the strongest part.

Yes, for two of our failures. Attention Tracker [2], which scores how much selected attention heads attend to a fixed instruction, shares them: its carried threshold breaks on two of four other layouts, and it fires on how-to steps. It does catch the third attack source on its own layout.

Bracketed ranges in these replies give 99.4% intervals for the confirmatory statistics of our second protocol, hashed before its runs, and 95% intervals elsewhere (verdicts in Table 27); a bracketed single number points to the References block at the end of the comment.

With its released heads on Mistral-7B v0.3, the detector passed our check against its published result. On its own layout, at a threshold set on clean documents alone, the share of how-to steps (added WikiHow steps [3]) it fires on is 0.88 above its share of clean passages. There it misses no attack from InjecAgent [4], Section 4.4's third attack source.

Its own layout's threshold, set on labelled pairs and carried to four others, breaks after a separate turn and with the scripted history (error 1.00 in both samples). On the tool and JSON layouts every interval contains 0.10. With test labels, some threshold has error at most 0.06 on each of the four (Table 4, Appendix G).

Weakness 1, continued. probes with different readout mechanisms.

We tested two, and neither removes the failure. With third-source strings in its fit, the linear combined refit catches nearly all held-out strings of that source (Figure 3a), so adding the source helped on that source where changing the readout did not. Based on this, we recommend testing any readout, linear or not, on an attack source that no fit or threshold used, as step 5 of our checklist says (Section 5).

The both-sources refit is a linear readout fitted on BIPIA [5] and human-written attacks [6], with harmless edits as negatives, and the combined refit adds how-to steps to its labels and threshold (Sections 4.3 and 4.4). An RBF support vector machine and an MLP, fitted on the combined refit's data with their own worst-kind thresholds (which minimise the worst error on any kind), miss 0.62 [0.52, 0.72] to 0.92 of third-source attacks, averaged over two shifted interfaces. This confirmatory result holds: these two nonlinear families, at this budget, do not rescue the refit (Section 4.6). Where one misses fewer attacks than the linear refit, it fires more on how-to steps, a threshold trade, not a rescue.

Three details set the scope of our Attention Tracker results, a carried threshold that breaks on two of four other layouts and firing on how-to steps (Appendix G). Its released threshold was set on short texts and fires on most clean passages of its own layout. We attribute this to passage length, not the interface, and the locked comparison already uses a threshold set on these passages. Its instruction sits in a system message on its own layout and in a user turn elsewhere. Reselected heads with a new threshold per layout pass the attack-versus-clean test in 7 of 8 cells; we read this repair on that test alone.

Q1, continued. You ask whether the failures occur in probes beyond TaskTracker that use different representations or detection mechanisms.

Yes, for new kinds of text. In an exploratory run, PIGuard [7], a text classifier, shows both failures: it misses a new attack source and fires on how-to steps. It reads only the document, so it cannot see the interface. So PIGuard answers the half of Q1 about new kinds of text. For the half about interfaces, Attention Tracker, which reads attention heads, has a carried threshold that breaks on two of four other layouts (Table 4, Appendix G).

At its shipped threshold, PIGuard misses 0.62 of third-source attacks and fires on 0.34 of how-to steps, so the same harmless kind breaks a different class of detector. On the two sources our refits were fitted on, it does well. It misses 0.060 of human-written attacks and none of 25 BIPIA strings, which bounds that miss rate at 0.113 (Section 4.4, Appendix G). A small test on BIPIA strings came before the protocol and informed these predictions, as the protocol's ledger records.

An earlier read of another released guardrail, in Supplement S47, failed its positive control, so it does not bear on interfaces. Section 2 cites one outside result [8]. There an attention detector falls to near chance on tool outputs from AgentDojo [9] in most model families, where data and format change together.

AuthorsOpenReview ↗

Reply part 2

Weakness 1, continued. test at least one substantially different probe.

We ran one, PIShield [10]. It shares our how-to failure, and our refit on its hidden states has the added-sentence failure at its training threshold. The check's two protocols fixed the predictions before any score existed and declared it exploratory (Appendix H).

PIShield reads a hidden state of Llama-3.1-8B-Instruct given the document alone. It catches nearly all BIPIA attacks and fires on no clean document. Against our prediction, it misses 0.236 of human-written attacks. It fires on most documents with a how-to step, and no threshold keeps both misses and these false alarms at or below 0.10. It rarely fires on documents with an added sentence from another article.

Our refit on its states, with no interface change, passes the attack-versus-clean test. At its training threshold it fires on 0.404 and 0.816 of added sentences. At the held-out threshold, the refit fitted on BIPIA attacks misses human-written attacks and fires on most how-to steps (Appendix H).

On a whole prompt, which it is not built for, PIShield fires on every clean document. A second test, also fixed in advance, delivered the document alone in four wrappers. A tool message broke its threshold in sample 2, narrowly (misses 0.132 [0.095, 0.180]), and no other wrapper did. All four wrappers kept the how-to failure.

Weakness 2 and Q2. The two shifts cover few real agent interfaces, and what about other chat templates, tool messages, structured tool outputs and several retrieved documents.

Thanks for raising this question. The revision adds five layouts, a trained tool role, a tool header and real tool content (Section 4.6, Table 4, Appendix G), and Appendix I adds a system role and a second trained tool role.

The five layouts are JSON tool output in a user turn, XML tags, the document in an assistant turn, three documents, and twelve earlier exchanges. The probes subtract a reference, the hidden state of a prompt with no document. We build it three ways (an empty document slot, a slot saying "No document was found.", and the task alone) and count a failure only when all three agree.

At the frozen threshold, error exceeds 0.10 under all three reference prompts in 10 of 20 cells, and is at most 0.10 under all three in none. The other 10 cells depend on the reference, and we leave them out of the count. We read Mistral-7B v0.2 at 0.5, chosen after the toolkit's 0.9 failed on its own interface; at 0.9 the count is 13.

The score also fails in 6 cells: with an empty slot, no threshold on the frozen score has error at most 0.10 there (point estimates, Appendix G).

AuthorsOpenReview ↗

Reply part 3

Weakness 2, continued. whether the same threshold- and score-level failures persist across these interfaces.

Yes, and they persist in the form you singled out, a repair of benchmark performance that does not recover detection. On the five layouts the revision adds, as on the submitted ones, passing the attack-versus-clean test does not show that a refit works, as the next three paragraphs show. A pass shows only that a repair works on the kinds of text it was tested on. Based on this finding, we propose accepting a repaired probe only after counting its misses and false alarms at its deployed threshold on harmless edits, and on harmless kinds and attack sources that neither its fit nor its threshold used. Without a refit, the frozen threshold fails under all three reference prompts in 10 of 20 cells and passes under all three in none (the other 10 depend on the reference; Mistral-7B v0.2 read at 0.5, and at the toolkit's 0.9 the count is 13). In 6 cells, with an empty slot, no threshold on the frozen score keeps error at or below 0.10 (point estimates, Appendix G).

Refitted on each layout, the both-sources refit passes the attack-versus-clean test in all 20 cells (point estimates). Yet the combined refit, refitted the same way, misses 0.55 to 0.97 of third-source attacks in 8 of 10 layout and model pairs (Section 4.6). This confirmatory result holds. The other two pairs are Phi-3-medium with JSON tool output and with three documents.

The training threshold's failure persists too. A later test, locked with our own timestamp, read these 20 cells again. At their training thresholds, the readout refit, with no harmless edits in its fit, and the hard-negative refit, with them as negatives, keep added-sentence false alarms at or below 0.10 in only 1 and 4 of 20 cells. A linear readout with more dimensions than training examples can separate its training set [11], so training pushes harmless training scores down and a threshold just above them sits too low for new text. Based on this, we propose setting the threshold on held-out scores of clean documents and harmless edits, never on training scores. We chose this rule after seeing the counts on the first interfaces, and this locked test confirmed that it stops added-sentence false alarms. Thresholds set on held-out scores of the edits keep added-sentence false alarms at or below 0.10 in 19 of 20 cells each, though the readout refit then passes the four-rate test in 15, short of the 16 we predicted (Section 4.2, Appendix E). So the rule stops added-sentence false alarms; it does not fix how-to steps or new attack sources.

On JSON tool output the both-sources refit also fires on 0.230 (Phi-3-medium) and 0.680 (Mistral-7B) of how-to steps (Appendix G).

All these layouts keep SQuAD passages [12]. JSON escaping changes BIPIA's code blocks, and the three-document and history layouts add other SQuAD text. Each refit here is one fit.

Weakness 2 and Q2, tool messages. Deployed agents use tool messages and arrangements of system, user and tool messages.

We tested two trained tool roles, one in your arrangement. On Llama-3.1-8B-Instruct, a test locked before its runs, with our own timestamp, puts the task in the system role, then a user request and a retrieval call, then the document in the tool role. All five predictions held. The both-sources refit carried from the model's own layout breaks (error 0.95 and 0.93). Refitted there, it passes the attack-versus-clean test yet fires on 0.22 and 0.14 of how-to steps, and the combined refit misses 0.29 of third-source attacks (Appendix I).

The released probes' templates reject or drop a tool message (Supplement S54), so we also used Mistral-7B v0.3. We carried the both-sources refit, with its threshold, from three user-turn layouts to two tool layouts. Where a reply and the question follow the tool message, its error is 1.00 and 0.99. With the tool message last, its error in sample 2 is 0.17 [0.06, 0.30]; there it hides no how-to failure, but the combined refit, carried the same way, misses 0.77 of third-source attacks.

On v0.3 the role, the declared-tools block and, with the tool message last, the token we read change together, so this tests the whole tool set-up. v0.3 has no released probe, and the carried threshold rests on the empty-slot reference only (Table 4, Appendix G).

Reviewer_v67WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of hidden-state-based prompt-injection probes under interface shifts. Experiments on two TaskTracker probes show that interface changes break the original thresholds in 11 of 12 settings, while refitting the linear readout repairs all 11. However, successful repair can conceal false positives on harmless text and poor generalization to unseen attacks. Adding diverse examples improves performance on known text types, but the repaired probes still miss 57–90% of attacks from a new source.

Critical strengths
  1. Practical relevance. The paper highlights that deployed prompt-injection probes can silently fail after routine interface changes, and that passing injected-versus-clean tests after repair does not guarantee reliability. This provides useful insights for real-world probe deployment.

  2. Clear diagnostic framework. Inspired by signal detection theory, the paper distinguishes failures at the threshold, frozen score, and feature levels. This simple framework helps determine whether threshold recalibration or readout refitting is needed and may generalize to other probes.

Critical weaknesses
  1. The evaluation covers only two probes from TaskTracker, using a single layer and the last-token representation. All documents are SQuAD passages with attacks appended at the end, and the interfaces are artificially constructed. Realistic settings involving JSON tool outputs, multiple retrieved documents, or agent workflows are not evaluated, making the generalizability of the findings unclear.

  2. Training probes on multiple interface formats is a natural approach to improving robustness. Although Appendix B includes exploratory multi-interface training and leave-one-interface-out experiments, these are limited to the same three synthetic interfaces. A more systematic evaluation on diverse, unseen interfaces is needed to establish the practical significance of interface-induced failures.

  3. Each BIPIA sample contains only 25 distinct test attack strings, with substantial overlap between samples, while InjecAgent contributes only 62 distinct strings. The 12 evaluation settings are not independent, and some confidence intervals assume independent test items despite repeated attack strings. Although string-level bootstrap results are provided, the limited attack diversity and differences in attack goals and lengths weaken the conclusions.

  4. The observation that adding a text category improves performance on that category is largely expected under distribution shift. The distinction between threshold and ranking failures has also been discussed in concurrent work. Moreover, the analysis is limited to linear readouts and does not investigate whether hidden representations encode the intended recipient of an instruction. The paper provides no generalizable solution to failures on unseen attack sources.

  5. Labels are assigned based on how examples are constructed rather than observed model behavior, leaving the connection to actual injection success unclear. Several methodological choices, including threshold selection, were made after inspecting results. Furthermore, the claimed confirmatory protocols rely on internally maintained timestamps rather than independent preregistration. The paper is also excessively dense, which reduces readability.

Questions for authors
  1. Would training probes on a wider variety of interfaces, including different conversation lengths, role assignments, JSON tool outputs, and multiple retrieved documents, substantially mitigate failures on held-out interfaces? This is important for assessing the practical significance of the paper's central finding that interface changes alone can break deployed probes.
  2. Is the failure to separate attacks from new sources and harmless how-to instructions caused by hidden representations lacking information about the intended recipient of an instruction, or merely by the limitations of linear readouts? Would the conclusions change with nonlinear probes, different layers, or different token positions?
  3. Have the authors considered adaptive attackers who deliberately exploit interface shifts, particularly by spoofing conversation turns within retrieved documents?
Reviewer_o78XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether a hidden-state prompt-injection probe still works when the prompt format changes. The authors take the two released TaskTracker probes and keep the question, the document and the injected sentence identical, changing only how these are packed into messages. They find the probes stop working in almost every changed format, that refitting the linear readout restores performance on the usual attacks-vs-clean-documents test, but that the refitted probes still false-alarm on harmless text and miss most attacks from a source they were not fitted on.

Critical strengths
  • the paper provides interesting insights for Task
  • explanation for the misplaced threshold is the most valuable part: training data in ~4k dims is always separable, so a cutoff picked on training scores sits right on top of the clean training points and any harmless-but-different text lands above it. That lesson generalizes well beyond this probe.
  • i appreciate unusually honest execution: predictions written down before the runs, failed predictions reported, dropped conclusions listed, and a script that rebuilds every number. The TF-IDF baseline matching the probe on BIPIA is a good control most papers would have quietly skipped.
Critical weaknesses
  • my main concern is a novelty -- this is an analysis of one specific released artifact (TaskTracker) rather than a new method or a general result, and imho "this probe breaks when you change the prompt format" is not enough for ICLR.
  • not realistic for agentic use. Everything lives in the user role: all three layouts are user+assistant only and the document is always inside a user message, so there is no tool role anywhere and instruction-hierarchy separation is never tested. In real agents injection arrives as a tool-call response, which is the one interface change that actually defines the agentic setting, and it's the one the paper skips.
  • the paper jumps straight in assuming TaskTracker is a default prompt-injection defense; it is not --- it's one activation-probe paper among several, and most defenses people deploy aren't hidden-state probes at all, so I need a paragraph on why this probe and why its failure matters. The writing is also compressed past the point of readability: the abstract is a chain of clauses with bare numbers, "11 of 12 cells" appears before a cell is defined, and Tables 2-3 need constant cross-referencing.
Questions for authors
  1. Mistral's chat template supports a tool role. What happens when the document arrives as a tool message rather than a user message?
  2. Is "set the threshold per kind of text" anything more than "test on more kinds of text"?
4 comments
AuthorsOpenReview ↗

Reply part 1

Weakness 1. the paper is an analysis of one released artifact, TaskTracker, with no new method or general result, and you find "this probe breaks when you change the prompt format" not enough for ICLR.

The paper's contribution is what it teaches about repairing and accepting a probe, and the tests and rules we propose because of it. It is not a new detector. It is an acceptance test, a threshold rule with a locked confirmation, a three-level diagnosis of what broke, and a checklist (Section 5). Its lessons concern how probes are fitted and accepted, not one prompt format, because both failures also appear on the original interface, with no interface change (Section 4.2).

You single out the explanation of the misplaced threshold, and we've revised abstract and Section 1 to state it early. A linear readout with more dimensions than training examples can separate its training set whatever it holds. So training pushes the harmless training scores down, and a threshold just above them fires on new harmless text. The cause is general, so this can happen to any linear probe fitted this way; we measured it on these probes. Based on this finding, we propose setting a refit's threshold on held-out scores of clean documents and harmless edits, never on training scores. We chose this rule after seeing the counts, and a test locked in advance then confirmed that it stops added-sentence false alarms (Section 4.2).

The second lesson is about acceptance. The attack-versus-clean test holds only documents like the repair's own data, so it can pass for the wrong reason. A refit passes it again in every broken cell, yet at its training threshold it fails our stricter four-rate test in every shifted cell of samples 2 and 3 (Section 4.2). A readout fitted to fire on added text, with no attack in its labels, passes that test on point estimates in all 4 probe-interface pairs; on Phi-3-medium its miss bound, 0.119, is above our 0.10 bar (Section 4.2). Based on these findings, we propose accepting a repaired probe only after counting its misses and false alarms at its deployed threshold, not an AUC, on harmless edits and on harmless kinds and attack sources that neither its fit nor its threshold used. We release the harmless edits.

The third lesson is that a patch fixes only the kinds it adds. Adding how-to steps to the fit and the threshold gives the combined refit, which passes every kind in its data (Section 4.4). Yet refitted on five more interfaces, it misses 0.55 to 0.97 of third-source attacks (from InjecAgent) in 8 of 10 interface-model pairs, a statistic locked before its runs that holds (Section 4.6). A later locked test finds that the how-to steps which fix that kind raise these misses (Section 4.4). In our main settings, no threshold on the combined refit, or on the both-sources refit before it, keeps both third-source misses and how-to false alarms at or below 0.10. Based on this, we recommend testing every attack source and harmless kind again after each patch, and calling a probe unfit for content where no threshold keeps both within the bar (Section 5).

We also see these failures beyond one model and one detector. Two later locked protocols find them on two Llama models: the both-sources refit passes the attack-versus-clean test yet fires on how-to steps, and the combined refit misses third-source attacks. In exploratory runs, how-to steps also break three other detectors: PIGuard, a text classifier; Attention Tracker, which reads attention; and PIShield, another group's probe (Appendices G and H).

Weakness 2 and Q1. the document always sits in a user message, so the paper never tests the tool role through which real agents receive injections. What happens on Mistral's template when the document arrives as a tool message.

We agree and thanks for the feedback. The revision tests two trained tool roles: Mistral-7B-Instruct-v0.3's (Section 4.6, Table 4) and Llama-3.1-8B-Instruct's, where all five predictions held. We used v0.3 because the template of v0.2, the released probe's model, raises an error on any tool message. Supplement S54 corrects our earlier claim that v0.3 lacks a tool role.

The both-sources refit is fitted on BIPIA and human-written attacks. We fitted it on three user-turn layouts and carried it, with its threshold, to the tool role. Its error is above the 0.10 bar in all 4 tool cells. When an assistant reply and a user question follow the tool message, it breaks in both samples. When the tool message comes last, sample 2's 99.4% interval reaches 0.06, and our locked test counts a break only when every interval clears the bar. Resampling articles gives the same.

Refitted on the tool layouts, the both-sources refit is usable in all 4 cells on point estimates, one at exactly 0.10. Yet refitted with the tool message last, the combined refit, which also has how-to steps in its data, misses 0.50 of third-source attacks (Appendix G calls the two refits A and D).

AuthorsOpenReview ↗

Reply part 2

Q1 and Weakness 2, continued. What happens when the document arrives as a tool message, and the paper never tests instruction-hierarchy separation.

We tested both, under three protocols locked before their runs (Appendix I). Fourteen of sixteen predictions held.

On Llama-3.1-8B-Instruct, the task sits in the system role and the document returns in the trained tool role, after a retrieval call. It has no released TaskTracker probe, so we read refits. The both-sources refit fitted there passes the attack-versus-clean test (error 0.01 and 0.06), yet fires on 0.24 and 0.18 of added sentences (training threshold) and 0.22 and 0.14 of how-to steps, and the combined refit misses 0.29 of third-source attacks. Fitted on the model's own layout and carried with its threshold, the both-sources refit breaks (error 0.95 and 0.93). All five predictions and the declared claim held.

On Llama-3-8B-Instruct, with the task in the trained system role and the document in a user turn, the released probe breaks at 0.5 (error 1.00 and 0.99, empty slot). The both-sources refit passes, yet fires on 0.63 and 0.62 of how-to steps, and the combined refit misses 0.73 of third-source attacks. Added sentences drew only 0.06 and 0.08, so that prediction and the declared claim failed.

On Llama 3.1's own layout, the how-to and third-source rates are higher (0.77 and 0.85; 0.56) and the added-sentence rate lower (0.10 and 0.04).

The third protocol keeps the Llama 3.1 tool layout and changes only the role word in the header of the turn that returns the document: ipython, the trained tool role, becomes user, a trained role, or tool, a header the official template never writes. Each new word is one token and ipython is two, so later tokens move one position. Five of six predictions held. Under both words, the threshold carried from the model's own layout breaks (0.92 to 1.00), the refit fitted on the new layout passes the attack-versus-clean test (under user exactly at the bar, 0.10, in sample 3), and the added-sentence and third-source failures remain (third-source misses 0.48 under user, 0.34 under tool). The how-to failure remains under tool (0.25 and 0.29) but not under user (0.06 and 0.05), so that prediction and the declared claim failed. On point estimates, a user header removes the how-to failure in this layout, and the other failures stay. We test no agent loop.

Q1, continued. What happens when the document arrives as a tool message rather than a user message.

We tested this on a released probe as well, and it fails under a tool header. Llama-3-8B-Instruct's template writes any role name into its header, so its released TaskTracker probe can meet a tool header. We read it at threshold 0.5 (Table 4; Appendix G).

A TaskTracker probe subtracts the hidden state of a reference prompt. We build it three ways: with an empty document slot, with the slot holding "No document was found.", or with the task alone. Under the tool header the probe's error is 0.99 in both samples with the empty slot, and at least 0.67 under every reference. With the empty slot, no threshold on its score works.

On its own layout the same probe already has error 0.14 in sample 3, above our bar.

Llama 3 has no trained tool role, so this tests a released probe under a tool header, not a learned role. Comments 2 and 3 give the trained tool roles of Mistral-7B-Instruct-v0.3 and Llama-3.1-8B-Instruct.

On v0.3 the role, the declared-tools block and, with the tool message last, the read token change together, so this tests the whole tool set-up, not the role word alone. There the carried both-sources refit fires on 0.02 of how-to steps; its threshold rests on the empty-slot reference only.

A later locked test renames only the document's header in Llama-3.1-8B-Instruct's tool layout (Appendix I). Under user the how-to failure goes, while the carried-threshold, added-sentence and third-source failures remain. Fitted under user and read under the trained ipython header, the both-sources refit has error 0.53 and 0.59.

AuthorsOpenReview ↗

Reply part 3

Weakness 2 and Q1, continued. In real agents, the injection arrives as a tool-call response, and you see this as the interface change that defines the agentic setting.

We agree, and we test tool output in a user turn in two ways (Table 4, rows 1 and 4), because Phi-3-medium's template drops tool messages.

First, JSON tool output. A main statistic of our second protocol, fixed before its runs, covers five new layouts, this one among them. Refitted on each, the combined refit misses 0.55 to 0.97 of third-source attacks in 8 of 10 interface and model pairs, and the statistic holds (Section 4.6). Phi-3-medium on JSON tool output is one of the two exceptions.

On this layout the released probes' frozen threshold is usable in none of the 4 cells under all three references; Phi-3-medium in sample 2 passes under some. Refitted there, the both-sources refit is usable in all 4 cells on point estimates.

Second, closest to your point, an exploratory test wraps the same third-source strings in InjecAgent's tool responses. This real tool content breaks the frozen threshold on both models: with the empty-slot reference, averaged over samples 2 and 3, its error is 0.36 on Phi-3-medium and 0.30 on Mistral-7B. Refitted there, the combined refit misses 0.387 of third-source attacks on Mistral-7B and 0.105 on Phi-3-medium, whose 95% interval spans the 0.10 bar (Appendix G).

Weakness 3. The paper assumes TaskTracker is a default defence, while it is one activation-probe paper among several and most deployed defences are not hidden-state probes. Why this probe, and why its failure matters.

Thanks for the suggestion, and Section 1 now has a paragraph, "Why this probe and these settings". We chose TaskTracker because its probes are public, so we test probes we did not build. Some of its authors ran it as a defence in the LLMail-Inject challenge, and later detectors compare against it. Its failure matters for two reasons. First, probes are cheap first screens before a slower check, and Gemini already runs probes for misuse (Kramár et al., 2026). A first screen needs few misses and false alarms at one threshold, which an AUC does not show. Second, the repair failures also appear on the original interface, so they do not need the interface change, and the usual acceptance test hides them. So the lessons can reach other probes refitted this way. Based on this, we propose judging any such probe by its misses and false alarms at its deployed threshold, on harmless text and attack sources outside its data. Section 2 covers text, attention and internal-state detectors, and we test all three kinds.

You also found the writing compressed past readability, with an abstract of chained clauses and bare numbers, "11 of 12 cells" before a cell is defined, and Tables 2 and 3 that need cross-referencing.

Thanks for pointing this out and we have revised the main text. The abstract now states our findings with few numbers. Section 1 defines a cell before counting cells, and Table 1 lists the nine kinds of text. Table 3 replaces the submitted Tables 2 and 3, one row per added kind; the full tables are Tables 24 and 25 (Appendix E).

Q2. Whether "set the threshold per kind of text" is anything more than "test on more kinds of text".

Yes, though the data matter more than the rule. The rule sets one threshold, not one per kind, because a document's kind is unknown when it arrives. It minimises the largest error rate over the listed kinds (Sagawa et al., 2020; Diana et al., 2021; Section 2). We renamed it the worst-kind threshold.

Over 20 random splits in each of 4 probe-interface pairs, the combined refit passes on every kind in its data in 78 of 80 fits with the worst-kind rule and in 68 with a pooled threshold, set on all harmless documents. With how-to steps only in the threshold, 25 of 80 pass, fewer in every pair, as our locked prediction said. With how-to steps only tested, none do. So a kind helps most in the readout's labels and its threshold; the worst-kind rule adds a smaller gain. A kind outside the list gains nothing and can lose: in a later locked test on five more interfaces (own timestamp), adding 4 how-to steps raised third-source misses at the worst-kind threshold by over 0.10 in 18 of 20 cells (Section 4.4).

By 95% Wilson upper bounds, with items treated as independent, 78 and 68 become 49 and 44 (Appendix E). Over bootstrap draws, the pooled rule's pass is fragile, while the worst-kind rule's survives nearly every draw (Section 4.4).

AuthorsOpenReview ↗

Follow up with the reviewer o78X

Dear Reviewer o78X,

As the author response period draws to a close, we would be grateful if you could review our responses and let us know whether any concerns remain. If our clarifications have addressed your concerns, we would appreciate it if you could reconsider your assessment accordingly.

Thank you again for your time and thoughtful review.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.