acceptodds
Under review as a conference paper at ICLR 2027

Teaching Agents to Defend: Experience-Augmented Execution Control against Indirect Prompt Injection

Abstract

Indirect prompt injection (IPI) has become a critical security threat to LLM agents, where adversaries embed control payloads in tool outputs to hijack agent actions. Existing defenses primarily address this threat by detecting or suppressing external instructions, or by constraining execution within predefined boundaries. However, these approaches face two fundamental limitations. First, suppressing external instructions conflicts with LLMs’ inherent instruction-following capabilities. Second, initial user tasks may not specify all instructions required for complex tasks, while predefined execution boundaries struggle to accommodate open-ended workflows. To address this dilemma, we present ActAuditor, a new defense paradigm that shifts IPI defense from controlling what an LLM may heed to controlling what an agent may do. Our key insight is that the natural-language understanding and instruction-following capabilities that expose LLM agents to IPI can also be harnessed for defense. Rather than restricting an agent’s ability to follow instructions, ActAuditor teaches it to explain and audit its own proposed actions without additional training. Technically, ActAuditor combines provenance-aware action attribution to establish the provenance of proposed actions, experience base with structured semantic features to provide relevant few-shot exemplars, and experience-augmented action audit to assess action security before execution. Experiments demonstrate that ActAuditor achieves 0% ASR under static attacks with negligible utility loss on AgentDojo. More importantly, under adaptive optimization-based attacks and on open-ended tasks, ActAuditor achieves the best security-utility trade-off among the evaluated defenses, while baselines suffer from low security or over-defense.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agent, Prompt Injection Attack, Defense Paradigm

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1UNXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActAuditor, a runtime defense against action-level indirect prompt injection in tool-enabled LLM agents. The key idea is to move the security boundary from controlling what external instructions an LLM is allowed to process to controlling which proposed actions an agent is allowed to execute.

For each proposed tool call, ActAuditor requires the agent to produce a structured provenance record identifying the operative instruction, the sources of the call arguments, and a rationale. Calls attributed directly to the trusted user task can be deterministically allowed, while calls driven by external instructions require additional auditing. ActAuditor-full maintains an experience base of previously audited externally induced actions, represented using structured semantic features and trusted allow/block labels. At runtime, similar experiences are retrieved and supplied to an LLM auditor, which returns allow, block, or defer.

The paper evaluates ActAuditor on AgentDojo and AgentDyn with multiple backbone LLMs and a large set of baselines. ActAuditor-base achieves 0% ASR against the evaluated static attacks, and ActAuditor-full improves the security–utility trade-off on open-ended AgentDyn tasks by permitting legitimate externally induced actions. The system also performs strongly against the evaluated adaptive attacks.

Critical strengths

The paper presents a strong conceptual reframing of indirect prompt-injection defense. In open-ended workflows, external instructions are not inherently malicious, so simply suppressing or rejecting all externally induced actions creates a poor security–utility trade-off. Auditing concrete proposed actions and their authorization relationships is a natural and practically meaningful alternative.

The provenance design is thoughtful. In particular, separating the source of the operative instruction from the source of tool arguments avoids conflating external data used to instantiate an authorized action with external instructions that authorize a new action.

I also find the role-level normalization idea interesting. Presenting the original user task and subsequent observations at comparable conversational roles reduces attribution bias while preserving the user task as the trusted root in the runtime.

The distinction between ActAuditor-base and ActAuditor-full is experimentally useful. AgentDyn clearly demonstrates why blocking all externally induced calls can achieve security at the cost of essentially zero utility, while experience-augmented auditing recovers substantial utility.

The experimental coverage is broad. The paper evaluates multiple LLM backbones, closed- and open-ended benchmarks, many baseline defenses, ablations, parameter sensitivity, human auditing, efficiency, and adaptive attacks. The adaptive attack evaluation is particularly valuable because several strong static defenses degrade substantially while ActAuditor remains near zero ASR.

Finally, the paper is unusually transparent about limitations, including the dependence on model capability and relevant experience availability, as well as the exclusion of output-only and data-integrity attacks.

Critical weaknesses
  1. My main concern is the fairness of the comparison between ActAuditor-full and training-free baselines on AgentDyn. Although ActAuditor does not update model parameters, its experience base is populated during cold-start rounds using human-validated audit decisions. Subsequent autonomous evaluation therefore benefits from domain- and benchmark-specific labeled precedents. The competing training-free defenses do not appear to receive an analogous set of supervised examples. I therefore do not think the strongest AgentDyn comparison should be interpreted simply as one training-free defense outperforming other training-free defenses. Rather, the result demonstrates the value of experience-augmented, human-seeded runtime adaptation. Relatedly, the cold-start protocol changes task arguments and wording to prevent exact matches, but this does not establish generalization to genuinely unseen workflows. The retrieved cases may remain close semantic neighbors from the same task family. A task-template or workflow-level held-out evaluation would substantially strengthen the generalization claim.

  2. A conceptual concern is that the provenance record is generated by the same LLM that is exposed to the adversarial observation. Thus, “provenance-aware attribution” is not equivalent to trusted causal provenance tracking; it is a model-generated attribution grounded by trusted source labels. In particular, the deterministic layer directly allows calls attributed to the trusted user task. This creates an important attack target: induce a malicious call while causing the model to falsely attribute the operative instruction to the trusted root. I would like to know whether the adaptive attack explicitly targets this failure mode.

  3. The experience base is assumed to be trusted and uses human-verified labels. Since ActAuditor-full depends heavily on retrieved precedents, the process by which experiences are curated, updated, and protected becomes part of the security boundary. Experience poisoning or stale/incorrect precedents are not currently studied.

The method also incurs nontrivial deployment cost. The reported full system has approximately three times the latency and input-token consumption of the undefended agent, with tens of seconds of end-to-end latency per task. This is acceptable for some applications but potentially significant for interactive agents.

  1. The scope is narrower than general indirect prompt injection: the method specifically targets attacks that manifest as malicious tool behavior and does not address output-only manipulation or factual/data-integrity corruption. The paper acknowledges this clearly, but the headline claims should consistently preserve this distinction.
Questions for authors

How should the comparison with training-free baselines be interpreted when ActAuditor-full uses a human-seeded experience base constructed from prior AgentDyn interactions? Can the authors evaluate with an experience base built only from disjoint task families or provide baselines with access to the same labeled historical examples?

Can the authors evaluate generalization to previously unseen workflow/task templates rather than only changing arguments and wording of related tasks across rounds?

Does the adaptive attack explicitly attempt to forge action provenance, especially by inducing a malicious action while causing the model to cite the trusted call 0 user task as its operative instruction? Can the authors report this failure mode separately?

How would ActAuditor handle poisoned, stale, or conflicting entries in the experience base, and what operational assumptions are required to keep this base trusted?

Given the reported ~3× latency/token overhead, can auditing be selectively applied to consequential or externally induced actions to reduce deployment cost?

Reviewer_GtCJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ActAuditor, a runtime defense against indirect prompt injection in tool-using agents. For each proposed tool call, the agent reports the source of the motivating instruction, the sources of its arguments, and an accompanying explanation. A deterministic check then permits calls attributed to the user task, while calls attributed to external observations are passed to an LLM audit that draws on retrieved and human-verified cases. The authors evaluate a base variant primarily on AgentDojo and a full variant primarily on AgentDyn. They report near-zero attack success rates under both static and adaptive attacks, while preserving useful task completion on open-ended tasks.

Critical strengths
  1. The paper studies the important and underexplored setting in which legitimate and malicious instructions coexist in external content, making the security–utility tradeoff central to the problem.

  2. The evaluation compares against 15 baselines spanning four defense paradigms, providing unusually broad context for assessing the proposed method.

Critical weaknesses
  1. The defense depends heavily on model-generated provenance: verifying a quotation does not verify that the user authorized the action, and the paper does not test attacks targeting false attribution.

  2. The method combines established elements with limited technical novelty(prompted action attribution, case retrieval, and LLM auditing) its main baseline comparison uses one backbone, and its experience-based results are tested on familiar task families.

Questions for authors
  1. ActAuditor assumes that the provenance the agent reports for each tool call is faithful. How often does the agent incorrectly attribute an externally induced call to the user task? Please manually assess a sample of provenance records and test an adaptive attack that explicitly induces plausible but false citations to the user task.
  2. How does ActAuditor-full perform when experience-base construction and evaluation use disjoint user tasks or task families? This would test whether retrieved precedents transfer beyond the repeated benchmark templates.
  3. Could the authors report how many human-confirmed cases were used to build each experience base? Comparisons with no retrieved cases, randomly selected cases, and retrieved cases without rationales would help show how much of the improvement comes from structured experience, rather than additional LLM judgment or human supervision.
  4. How does ActAuditor differ from Task Shield [1] and AlignmentCheck [2], which also audit agent actions against user goals?

[1] The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

[2] LlamaFirewall: An open source guardrail system for building secure AI agents

Reviewer_FW9hOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses defense against attack scenarios such as indirect prompt injection (IPI). Existing approaches have either relied on the model itself to recognize prompt injection attacks or blocked disallowed actions at the system level. However, the former remains vulnerable to evasive attacks, while the latter also rejects legitimate external instructions, degrading utility. To address this, the paper has the agent structurally record the sources that motivate each action before it is executed. Based on this provenance and relevant past precedents, the agent then decides whether to execute the action, block it, or defer it to a human for judgment. As a result, the paper achieves effective defense against indirect prompt injection with negligible utility loss.

Critical strengths
  1. The proposed method, ActAuditor, combines recording the provenance of each action with making decisions based on past precedents. This combination is a well-motivated design, and it is effective in the evaluated settings.

  2. The paper presents a promising approach to defending against indirect prompt injection attacks delivered through tool outputs, a setting in which many existing approaches have struggled.

Critical weaknesses
  1. The most critical assumption underlying this method is that the model can accurately determine the provenance of its own actions, yet the experimental evidence for this assumption appears weak. The paper would benefit from a more direct analysis and dedicated experiments on this point. In particular, production-grade CLI agents today operate over very long histories with highly complex interactions, and it is unclear whether the model can reliably determine and record the provenance of its actions in such settings. Moreover, the current recording format itself appears to have inherent limitations in handling this level of complexity.
Questions for authors
  1. Could the authors directly measure how accurately the model attributes the instruction source of each action, rather than relying only on end-to-end ASR and UA?

  2. How does attribution accuracy change as interaction histories become longer and more complex, as in production-grade agents?

  3. Has the method been evaluated against an adaptive attacker that explicitly aims to induce misattribution?

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. As the Author–Reviewer Discussion period is closing soon, please actively engage with the authors, consider their responses, and update your final scores and comments.

Best regards,

Area Chair

AuthorsOpenReview ↗

Author Final Remarks by Authors

Dear Senior Area Chairs, Area Chairs, and Reviewers:

Summary of Strengths Identified by Reviewers
We sincerely thank the reviewers for thoughtful review and are encouraged by the clear consensus on our contributions:

  • Clear Novelty and Significance: Our work was thought to study "the important and underexplored setting"(GtCJ), present a "promising approach" (FW9h) and "a strong conceptual reframing" (1UNX) of IPI defense.
  • Broad Empirical Validation: Our evaluation compares against 15 baselines, providing unusually broad context for assessing the proposed method(GtCJ), with Reviewer 1UNX remarking "The experimental coverage is broad."

Addressing Critical Concern

The table lists each addition and where to find it. Changes in the revised PDF are in blue.

Concern (raised by)What was addedWhere
No experiments about the accuracy and robustness of action attribution (FW9h W1,Q1,Q2,Q3; GtCJ W1, Q1; 1UNX W2,Q4)New experiments to confirm the accuracy of LLM action attribution under two benchmarks and adaptive attacks. Accuracy≥96% under all configurations,ASR≤1.67% under adaptive attacks.Section 4.2, Appendix D.1,E.3,E.4; Table 1,8,19
No cross-domain generatizeability evaluation (GtCJ Q2; 1UNX W1,Q2)We added experiments evaluating cross-domain generalizability against two new baselines, and analyzed our superiority and limitations.Appendix D.5, G.2; Table 10
What design elements contribute to outstanding performance? (GtCJ Q3; 1UNX W1,Q1)We conducted more fine-grained ablation experiments. The results verify the importance of designed structured experience.Appendix D.2; Table 9
What are our innovations, and how does it differ from Task Shield and LlamaFirewall (GtCJ W2,Q4)We revised the main text to better position ActAuditor as a novel defense paradigm and clarify its innovations and challenges. We analyzed the differences between ActAuditor, Task Shield and LlamaFirewall, and validated ActAuditor’s superiority via direct comparison experiments.Section 4.2, Appendix A; Table 3, 10
How to maintain the experience base (1UNX W3,Q4)We discussed how to maintain the experience base and handle poisoned, stale, or conflicting entries.Appendix B
Can latency and token overhead be further improved (1UNX W4,Q5)We reiterated the applicable scenarios and discussed how selective auditing could reduce runtime latency and token consumption.Appendix D.9

Final Remarks

We appreciate the rigorous review which has significantly improved our work. All three reviewers focused primarily on the accuracy and robustness of action attribution, as well as the generalizability of our approach. The new experiments and the refinements we made to the paper have been acknowledged and praised by Reviewer 1UNX. We would be grateful if the reviewers and AC can reevaluate the paper based on the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.