acceptodds
Under review as a conference paper at ICLR 2027

Why Do Defenses Fail? Counterfactual Diagnosis of LLM Agent Safety

Abstract

A lower attack success rate (ASR) in LLM agents can reflect selective protection or impaired legitimate behavior. We introduce Three-Axis Safety Diagnosis through Counterfactual Measurement and Evaluation, or TASD-CME, a white-box protocol that tests a proposed explanation by pairing a prespecified diagnostic test with a targeted internal intervention at the same frozen decision point. The protocol asks whether external directives receive undue influence, harmful use can be suppressed while preserving benign capability, and safe actions are preferred when harm is recognizable. Local support requires a qualifying failure signature, verified intervention delivery, the intended target response, behavioral improvement, retained legitimate performance, and effects beyond paired controls. Across five attack families and two models, prespecified interventions lower first-action ASR point estimates in all ten settings, but only four satisfy this joint rule. For tool-chain manipulation, Gemma and Qwen reduce ASR by 58.00 and 50.53 percentage points, respectively. Qwen meets the joint requirements; Gemma's diagnostic comparison fails the prespecified balance check despite its larger behavioral gain. In Qwen, the intervention retains 98.0% of baseline legitimate-task utility and exceeds the random control's ASR reduction by 44.13 points. TASD-CME separates behavioral gains from support for their proposed local explanations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent security, indirect prompt injection, defense evaluation, counterfactual analysis, white-box diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rTZGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a lower attack success rate (ASR) does not, on its own, show why an agent resisted an attack. It proposes TASD-CME, a white-box protocol that assigns one functional explanation to each attack family in advance and tests it at a frozen critical decision point (CDP), just before the first attack-advancing action. There are three candidate relations: source authorization for indirect prompt injection (IPI), capability selectivity for intent hijacking (IH) and memory poisoning (MP), and safe-candidate priority for objective drift (OD) and tool-chain manipulation (TC). A model-family cell earns "local support" only if an E1 diagnostic signature is present, the mapped activation-level operator is delivered, moves its target readout, and lowers first-action ASR, and the effect stays within utility, damage, and over-refusal bounds while beating random, wrong-site, orthogonal, and matched controls. The operators are SAG (subtracting a fraction of the residual difference between an external directive and a length-matched data-like control), CSS (contracting selected units toward their calibration means), and SPS (adding a calibrated safe-minus-unsafe direction). An exploratory black-box screen (CPSep) on AgentDojo (GPT-4o, Llama 3.3) and AgentLAB (eight models) motivates the hypotheses, and the confirmatory study runs Gemma-4-12B-it and Qwen3.5-9B on five AgentLAB families. All ten mapped operators lower the first-action ASR point estimate, but only four cells (both IPI cells, Gemma/MP, and Qwen/TC) reach local support; the rest are associated effect (2), E1/E2 conflict (1), or inconclusive (3). The authors' takeaway is that a behavioral gain and support for its explanation can come apart.

Critical strengths

S1. The question is well chosen, and the evaluation follows a stricter procedure than is common in agent security. A mechanism claim must pass a diagnostic signature, a verified intervention, utility guardrails, and paired controls at the same decision point, with a held-out confirmation split, task-clustered intervals, and within-cell BH adjustment, and the authors report the cells that fail. The Qwen/TC refusal-bias control shows why this matters: it moves the SPS target readout by the same .650 margin but keeps only .873 utility with .107 over-refusal (SPS: .980 and .007). This procedural care is the main reason the paper is worth engaging with. As argued below, however, it is applied to interventions that cannot test the stated explanations, so it does not carry the paper to acceptance.

S2. The IPI versus OD contrast is the most informative result. In the eight-model screen, OD responds most strongly to Directive Removal (66% to 7%), yet its external-directive influence at the decision point is close to zero (DIE\mathrm{DI}_{E} = .010 and .023), while IPI shows positive DIE\mathrm{DI}_{E}, trusted and external influence statistically equivalent (TOST within ±.08), and a highly decodable source signal (probe AUROC .982 and .973). If it holds beyond two models, this is a useful warning that input-ablation results do not identify the internal failure.

Critical weaknesses

W1. The central claim is not supported. The interventions do not manipulate the hypothesized causes, so "local support" does not license the explanations it is attached to. This is the main reason for my rating, and it cannot be fixed without redesigning the interventions.

(a) Source authorization. The claimed failure is that the model reads where a directive comes from but does not gate on it. SAG does not test this. It subtracts part of h(xE,d)−h(xE,d0)h(x_{E,d}) - h(x_{E,d0}), i.e., part of the directive's own contribution relative to a data-like control. This is a content-level ablation, and it would lower DIE\mathrm{DI}_{E} and ASR whether the real problem is missing source gating, source misperception, or a general over-weighting of imperative text. Because SAG is built from the same paired contrast that DIE\mathrm{DI}_{E} measures, passing the L2 check is largely expected and says little about the hypothesis. The numbers are consistent with this: the PAS gain matches the drop in DIE\mathrm{DI}_{E} (Gemma: .181 to -.272, ΔPAS\Delta\mathrm{PAS} = .452; Qwen: .162 to -.258, ΔPAS\Delta\mathrm{PAS} = .420), so the trusted side barely moves. A real test would intervene on the source representation itself and show that directive influence and ASR follow. The E1 premise is also weaker than presented. Ye et al. (ICML 2026) show that models assign roles by how text sounds rather than by its tag, so a slot-label probe with AUROC .98 may simply be reading the tag or position, and the control-task and paraphrase checks do not rule this out. This work is not discussed.

(b) Capability selectivity. The evidence here is partly self-referential. E1 qualifies a capability constraint when the authors' own operator cannot reach .20 feasible suppression, and E2 then asks whether that same weak operator lowers ASR. Showing that a weak operator still helps does not show that harmful/benign entanglement explains the failure, and IH, where selective suppression is easiest (HS .379 and .420), ends up as conflict or inconclusive rather than as evidence against the hypothesis. HS is shared by E1 and E2, so the "joint" requirement is not two independent pieces of evidence. The constraint is defined on a small grid (two rules, four or five doses), as Appendix B.4 acknowledges, so it describes the intervention family rather than the model.

(c) No test against competing explanations. Each family is assigned one relation a priori, and the paper itself acknowledges that relations can overlap. The four controls only show that each operator beats a comparably sized perturbation. They do not show that the assigned relation explains the failure better than an alternative, which is exactly what a diagnosis must establish. The paper's own Table 3 argues against both capability families: the non-mapped SPS lowers ASR more than the mapped CSS in all four IH/MP cells (e.g., Gemma/MP: 25.33 vs 10.00 points), yet Gemma/MP is reported as local support for a capability constraint. Without cross-relation controls, the protocol cannot answer the question in its own title.

W2. The claims go well beyond the evidence, the evidence states are fragile, the related work is incomplete, and the paper is hard to read.

(a) No defense is diagnosed. The title promises an explanation of why defenses fail, but TASD-CME is never applied to an existing defense; the operators are the authors' own diagnostic interventions on undefended models. AgentLAB ships baseline defenses (Self-Reminder, Llama Guard, Repeated Prompt, a DeBERTa detector), and many more exist for IPI (spotlighting, StruQ/SecAlign, MELON, PromptArmor). The paper does not diagnose any of them.

(b) Narrow setting and no evidence of generality. The confirmatory system is a single tool-calling agent in AgentLAB's simulated environments, run on two roughly 10B open-weight models. Closed models appear only in the black-box screen, and no multi-agent system is tested, although Figure 2 includes agent-to-agent messages among the external sources. Only the first action is scored, within a window of at most 48 tokens, for attacks that AgentLAB runs over 5 to 20 turns, and there is no evidence that the reductions hold over full trajectories. Basic agent settings (scaffold, system prompt, reasoning mode, and the attacker that produced the frozen prefixes) are not reported. Most importantly, the conclusions do not replicate even across the two models tested: only IPI gets the same evidence state in both, and MP and TC reach local support in one model only. With two models that disagree, the title and abstract substantially overstate the scope. Each model also has 400+4×300400 + 4 \times 300 audit tasks while AgentLAB has 644 test cases in total, and the paper does not say how the extra tasks were built.

(c) Fragile evidence states. The abstract's headline contrast, Gemma/TC "associated effect" at -58.00 points versus Qwen/TC "local support" at -50.53, is decided by a one-scalar SMD gate, and the Gemma strata that fail it contain only 2 and 4 resistance tasks (Table 19). A conclusion featured in the abstract should not rest on so few tasks. Apart from the replay-grouping check, there is no sensitivity analysis for the ∣SMD∣≤.50|\mathrm{SMD}| \le .50, .20 difficulty, or .08 PAS bounds, or for the stratification. Some inconclusive cells look like floor effects (Qwen/IH has 5 recognized tasks at 3.33% Raw ASR). Several reported quantities are also unusually uniform or tight (Q5), and the "registered" and "frozen" claims cannot be verified from the submission.

(d) Positioning. I looked for prior work that would scoop the full protocol and did not find any, so the combination of a prespecified diagnostic signature, a targeted internal intervention, and paired controls at a frozen agent decision point appears novel. Its components, however, have close relatives that the paper does not cite. SAG belongs to a known family of internal IPI defenses (CachePrune, ICON), and activation-based detection of injected instructions is well studied (TaskTracker, Attention Tracker, PIShield, InstructDetector). Counterfactual attribution of agent actions also includes MELON and CausalArmor, besides the cited AgentSentry and AttriGuard. CPSep's Directive Removal closely mirrors PromptArmor, which already reports AgentDojo ASR below 1%, so the near-zero removal results are expected. Two concurrent preprints, which I do not count against the paper, should be discussed in a revision: Dong et al. (arXiv 2608.02657), who probe IPI exposure in agent hidden states (including Qwen3.5-9B), report a recognition-action gap close to the safe-candidate-priority hypothesis, and rank prespecified explanations of the probe signal; and V-Steer (Zeng et al., COLM 2026), which restores instruction hierarchy at inference time by editing cached value vectors.

(e) Presentation. The paper is very hard to read, and this is not a cosmetic problem: it makes the central claims hard to check. It introduces more than a dozen acronyms and estimands (TASD-CME, CPSep, CDP, SAG, CSS, SPS, E1/E2, L1/L2, DIE\mathrm{DI}_{E}, PAS, HS/BR/GD, MSU(rec)M_{SU}(\mathrm{rec}), ΔSPA\Delta_{SPA}, SMD) and twelve evidence states, and the core estimands are defined only in Appendix B. Some labels mislead, e.g., "underpowered" means a scalar-balance failure in Table 14. Text in the main figures (e.g., Figures 1 and 4) is too small to read at normal zoom, and the results figures and tables (e.g., Figure 5, Tables 3 and 4) pack the outcomes into abbreviations and short state codes that are hard to parse. The paper needs a substantial rewrite: define each readout and operator in one sentence at first use, walk through one cell end to end, and redraw the key results with plain labels.

Questions for authors

Q1 to Q4 concern the core soundness problems, and answering them would require substantial new experiments. Q5 and Q6 affect my confidence in the reported numbers. Q7 and Q8 are lower priority.

Q1. For each relation, what pattern of E1 and E2 results would falsify it? For capability selectivity, why does an ASR reduction under a constrained CSS point support a capability constraint, and why does IH (easier selective suppression plus an ASR reduction) not count against it?

Q2. Can you intervene on the source representation instead of the directive content, e.g., move the external-slot state along the source-probe direction toward and away from "trusted", or swap source-bearing residuals between the E and T slots, and show that DIE\mathrm{DI}_{E}, PAS, and ASR respond? How does SAG behave on legitimate tasks where the user asks the agent to follow instructions found in external content? How do you rule out that the source probe reads the role tag or position rather than perceived authority (cf. Ye et al., ICML 2026)?

Q3. Please report the full gate pattern (target response, utility, four-control specificity) for the non-mapped operators in every cell. Given that SPS lowers Gemma/MP ASR about 2.5 times more than CSS, why attribute Gemma/MP to capability selectivity rather than safe-candidate priority?

Q4. How do the evidence states change under other thresholds (e.g., ∣SMD∣|\mathrm{SMD}| at .40/.60/.75, difficulty at .15/.25, PAS bound at .05/.10) and other stratifications? What are the confidence intervals of the Gemma/TC mild and severe SMDs, and does the Gemma/TC versus Qwen/TC distinction survive?

Q5. Please explain: (i) why benign retention is exactly .960 and general damage .030 or .040 in all six CSS cells for both models (Table 11), and why the Gemma and Qwen MP grids have nearly identical BR and identical GD values (Table 18), i.e., whether these are computed separately per model; (ii) how E1 intervals only .003 to .01 wide (e.g., HS .379 [.377, .380]) arise from 150 to 200 tasks, while the ΔASR\Delta\mathrm{ASR} intervals are about 5 to 16 points wide; (iii) why SAG at dose .20 on one layer pushes DIE\mathrm{DI}_{E} far below the data-like control (to -.272 and -.258), when even full subtraction would naively bring it only to about zero; and (iv) how the 400 and 300 tasks per family were derived from AgentLAB's 644 test cases, and whether tasks share templates. Do the anonymized artifacts include per-task raw records sufficient to recompute the main tables?

Q6. Is there a timestamped external record (e.g., an OSF registration or a commit hash predating the confirmation runs) that verifies the "registered" and "frozen" settings?

Q7. Do first-action reductions persist over full AgentLAB horizons, and how is first-action success defined for tool chaining, where single calls are meant to look benign? Why does the CPSep AgentLAB screen use a uniform 7-turn cap and a 1-to-5 judge instead of AgentLAB's per-attack budgets and success criterion? Are the screen and confirmation models the same checkpoints (Qwen3.5 MP is 0% Raw in Table 8 but 16% in Table 2)? Which scaffold, system prompt, reasoning-mode setting, and attacker were used in the confirmatory runs, and are the Table 23 operator layers shared across both models?

Q8 (suggestion). Applying TASD-CME to at least one existing defense, such as AgentLAB's Repeated Prompt or DeBERTa detector, or SecAlign/PromptArmor on AgentDojo, would directly answer the title question.

Satisfactory answers to Q2 to Q5 would be the minimum for me to reconsider. Given how much new experimentation they require, I do not think the paper can reach the bar for ICLR in this cycle.

Reviewer_esmnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work proposes a way to diagnose why AI agents take unsafe actions, rather than measuring pure ASR. Using AgentDojo, the authors test three potential issues: agents trust untrusted instructions, harmful behavior cannot be suppressed without losing abilities, or that they favor unsafe actions even when that unsafeness has been detected (via probes). The authors pause agent rollouts right before a risky action is taken, modify the model's activations and/or inputs, and check whether safety improves. Across five attacks and two open-weight models, all proposed interventions reduce ASR.

Critical strengths

Ultimately this paper asks an interesting question, and does a good job of answering it. In seeking to understand how agentic defenses (against prompt injection, e.g.) fail, the authors begin by testing candidate defenses on black-box models, and use further failure modes to form hypotheses for how to steer models where one might have white-box access. As far as I am aware, using activation monitoring and steering is an underexplored part of defending agentic systems, and one that the authors do well to consider here. The experimental design is sound as well, with the authors replaying identical rollouts across multiple intervention points.

Critical weaknesses

This paper is almost impossible to parse / read / review for a person. It uses far too many acronyms, the figures try and report far too much information than they reasonably have room for, and the language is evocative of AI slop. I would prefer to read non-standard or poorly-written human English than this, which is why my review is rather lacking in rigor. I have read and understood enough to catch the high-level ideas and believe them to be worthwhile, but I have not been able to appropriately interact with the work given its lack of readability.

Questions for authors
Reviewer_LKHFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a drop in attack success rate doesn’t tell you why an agent resisted: it could be selective protection, or it could be that the agent got worse at legitimate work. The authors propose TASD-CME, which freezes a trajectory just before the attack-advancing action and tests one prespecified explanation at that point. The candidate explanations are: external directives get treated like trusted ones; harmful and benign capability are hard to separate; or the model prefers the unsafe continuation even when harm is recognizable.

Critical strengths

The core point is right and worth making. An ASR drop can come from selective protection or from breaking the agent, and benchmarks rarely separate the two. I also appreciate the discipline of fixing hypotheses and settings before confirmation. The refusal-bias control is a nice touch: it shows that a broad refusal shift can move the same readout as the targeted intervention while hurting utility, which is exactly the kind of confusion the paper wants to catch.

Critical weaknesses
  1. The paper is very hard to read. The main text is packed with acronyms and evidence states. What the three interventions actually do only becomes clear in Appendix B. After the main text I still couldn’t say what was intervened on or what a “signature” concretely looks like. One worked example carried through in plain language, and far fewer named states, would help a lot.
  2. The title promises a diagnosis of why defenses fail, but no existing defense is diagnosed. The ASR reductions being explained come from the authors’ own diagnostic operators. I expected something like taking instruction hierarchy or spotlighting and asking whether its gain comes from source gating or from blanket suppression. That would show what the protocol is actually good for.
  3. Target engagement is largely guaranteed by construction. SAG subtracts the activation difference between the directive run and a data-like control, then checks that directive influence goes down. SPS adds a safe-minus-unsafe direction, then checks that the safe-minus-unsafe margin goes up. For CSS, the suppression readout is literally shared between E1 and E2. So passing these checks mainly confirms the operator was applied, not that the hypothesized relation drives behavior. SAG also needs a paired run without the directive, which is close to the directive-removal oracle, so an ASR drop is expected whatever the real failure is.
  4. The protocol can’t tell explanations apart, because each attack family gets exactly one hypothesis. A diagnosis should pit hypotheses against each other in the same setting. The results already show SPS lowering IPI attack success, so the safe-candidate account also “works” there. Without running competing relations through the same checks, “support for source authorization” doesn’t rule anything out. 5. I’d suggest running every applicable relation in each cell and reporting which ones survive. The final evidence states rest on thresholds and proxies that feel arbitrary. The Gemma tool-chain case has the largest behavioral effect, but it is denied support because one recorded scalar fails a balance check. Similar cutoffs decide other cells. It’s unclear what these lines mean, and the headline split between supported and unsupported cases may partly reflect where they were drawn. A sensitivity analysis would help.
  5. The black-box screen feels like a separate study. It uses different models, and by design its results don’t feed into the white-box tests. Meanwhile, the white-box study itself is narrow: two mid-sized models, only the first action, and teacher-forced scores on fixed candidate pairs. Either connect the screen to the main study (e.g., run it on the same two models and check consistency) or shorten it.
Questions for authors

How would TASD-CME classify an actual published defense? What happens when a competing relation is tested in the same cell? How much do the evidence states change if the thresholds move?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.