- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces ATrust, a defense mechanism against indirect prompt injection (IPI) in LLM-based agents. The main idea is to progressively expand a task-specific trust domain during agent execution by admitting external information considered relevant to the user's task. The proposed mechanism uses a fine-tuned language model to decompose external observations into task-related and unrelated semantic segments. Task-related information is then incorporated into an accumulated set of trust anchors used to guide subsequent agent decisions. The paper includes an evaluation of ATrust on AgentDojo and AgentDyn against several existing defenses, reporting favorable security–utility trade-offs, particularly for long-horizon agent tasks.
Critical strengths
-
The paper addresses an important security problem for LLM agents: mitigating indirect prompt injection without unnecessarily preventing legitimate task execution.
-
The authors correctly recognize that external observations may contain both useful information and malicious instructions, and that rejecting entire observations can negatively affect agent functionality.
-
The evaluation includes multiple agent backbones and two benchmarks, as well as ablation studies and experiments involving adaptive attacks.
-
The reported improvements in utility under attack are interesting and suggest that selectively retaining task-related information may be a useful practical defense strategy.
Critical weaknesses
-
The novelty is low and in addition the paper does not correctly characterize existing security approaches. The key motivation of the work is that conventional defenses rely on static trust boundaries and that dynamically adapting these boundaries constitutes an important conceptual advance. I find this characterization not justified.Dynamic security decisions have been extensively investigated in the security literature. For example, attribute-based access control (ABAC) and context-aware authorization models allow decisions to depend on dynamically changing attributes, including time, location, environmental conditions, and system state. Usage control (UCON by Ravi Sandhu) models additionally consider continuously evaluated conditions and changes occurring during resource usage. Furthermore, decisions concerning whether information may be used in a particular context can depend on its provenance, integrity, source, and intended use. Security information and event management (SIEM) systems also dynamically assess security-relevant observations gathered from different monitoring and detection components.These approaches demonstrate that the general concept of evolving security decisions based on execution context and accumulated observations is already well established. The paper does not adequately distinguish its proposed mechanism from these established concepts. Indeed, Appendix A.3 acknowledges existing agent defenses that dynamically assign execution rights or enforce dynamic constraints, yet the paper continues to characterize existing defenses as fundamentally static. The authors should have clarified whether their contribution is intended to be a new trust model, a new information admission mechanism, or a particular implementation of context-aware filtering for LLM agents. The latter may be useful, but is a more limited contribution than the paper currently claims.
-
The work has a fundamental confusion between task relevance and trustworthiness. My most significant technical concern is the assumption that information relevant to the user's task can be admitted as a trust anchor. In Section 3.2, Equation (2), the model classifies semantic segments based on their relevance to the task. The subsequent equations incorporate related information into the accumulated trust domain. However, relevance is not a sufficient criterion for trustworthiness. Consider an agent tasked with paying an invoice. An attacker may manipulate the invoice's bank account information. The malicious account number remains highly relevant to the task, and its inclusion may be essential for completing the requested operation. Nevertheless, the information is not necessarily trustworthy. Similarly, a malicious instruction may be presented as a necessary step for completing the original task. Such an instruction may be entirely consistent with the apparent task objective while directing the agent toward an unauthorized action. The paper appears to assume that malicious instructions are generally distinguishable because they introduce objectives unrelated to the user's request. This is not a sufficiently general threat model for indirect prompt injection. The key problem here is that the authors do not distinguish task relevance, source trustworthiness, information integrity, and authorization to use information for specific actions. Without such a distinction, the rationale for promoting task-relevant information to the status of a trust anchor is not convincing.
-
The paper does not adequately specify the trust-boundary mechanism. The description of the proposed mechanism remains largely abstract. Section 3 introduces a sequence of semantic decomposition, relevance classification, information consolidation, and trust-anchor accumulation operations. However, several fundamental design decisions are insufficiently explained. More specifically: (i) What precise security semantics are associated with a trust anchor? (ii) How is the trustworthiness of an admitted item established? (iii) How does the mechanism distinguish task-related data from instructions that are authorized to control the agent? (iv) How are information provenance and source integrity represented? (v) What prevents an attacker from inserting malicious but task-relevant information into the accumulated trust domain? (vi) How is the semantic consolidation function implemented, and what ensures that it preserves the intended information? Appendix F presents a case study, but this case study does not resolve these questions. The example primarily illustrates the removal of an extraneous malicious instruction while retaining legitimate task information. A more detailed end-to-end example is necessary to explain the actual security decisions made at each step and to demonstrate how the proposed mechanism handles adversarial information that is closely aligned with the legitimate task.
-
The approach lacks well-defined security properties and guarantees. The mathematical formulation in Section 3 describes the mechanism but does not establish meaningful security guarantees. For example, Equation (3) introduces completeness, disjointness, and non-fragmentation requirements for semantic decomposition. However, it is unclear how these properties are guaranteed by the learned model or validated at runtime. More fundamentally, the paper does not define a security invariant characterizing which information is allowed to influence particular agent actions.The proposed framework does not establish properties concerning unauthorized actions, information-flow integrity, or the prevention of trust-boundary violations. I am not necessarily suggesting that formal verification is required for an empirical machine learning paper. However, a mechanism explicitly presented as enforcing trust boundaries should at least specify the security properties it intends to maintain and explain why its operations are expected to preserve them. The current formulation does not adequately connect semantic relevance classification to security enforcement.
-
The trust domain is monotonically expanded. Equation (5) defines trust-boundary evolution through the accumulation of newly admitted trust anchors. This suggests to me that the domain expands monotonically. However, trust and authorization decisions are not necessarily monotonic. Information may become stale, be superseded, lose validity, or subsequently be identified as compromised. Context changes may also invalidate previous assumptions. The paper does not adequately discuss revocation, expiration, conflicting observations, or restrictions on the scope and duration of admitted information. This limitation is particularly important for long-horizon agent tasks, which are a central motivation of the paper. The authors did not explain whether admitted information can be removed or reassessed and how they prevent previously admitted information from improperly influencing subsequent actions.
-
The valuation does not sufficiently investigate the fundamental failure cases. The experiments report promising improvements in attack success rate and utility, and the inclusion of adaptive attacks is welcome. However, the evaluation does not adequately address the central security questions raised by the proposed mechanism. In particular, the authors should investigate attacks in which malicious information is directly relevant to completing the task, such as manipulated payment details, fraudulent task-specific prerequisites, corrupted tool outputs, or compromised information from apparently legitimate sources. The adaptive attacks in Section 4.5 are relatively limited, especially the manually designed attack based on asserting that external information is trustworthy. More challenging attacks should exploit the actual admission criterion and attempt to contaminate the trust-anchor set over multiple execution steps. The experiments did not evaluate whether errors in trust-anchor admission propagate or accumulate during long-running tasks. Finally, the evaluation compares the method primarily against prompt-injection defenses. It does not establish how much of the reported improvement is attributable specifically to dynamic trust-boundary expansion rather than context-aware semantic filtering and the specialized training of the admission model.
Questions for authors
None
9 comments
Response to Reviewer 5ud2's W2
Response to Reviewer 5ud2's W2
We agree that semantic-level relevance is not a sufficient criterion for trustworthiness. ATrust does not assume otherwise.
1. Task relevance does not mean semantic-level or scenario-level similarity. In Eq. (2), task relevance means whether the specific information is needed to complete the user’s intended task. makes this judgment using the user’s goal and accumulated task evidence, together with the current tool call and its arguments. Mentioning the task or claiming to be a prerequisite does not automatically satisfy this criterion. Consider an agent tasked with paying invoices: an attacker might tamper with the bank account details within an invoice. This malicious account number is highly relevant to the task. A lot of tasks in the benchmarks evaluated in the paper already cover such scenarios and attack settings. However, agents are also need to retrieve the user’s personal information from user query, documents or other sources. Our proposed "trust boundary expansion" aims to preserve this user-related information, which is otherwise only accessible via external observation. Consequently, if an agent is instructed to use an attacker-supplied account, the trust anchor trace will contain evidence that enables the agent to recognize it as an attacker’s account rather than one belonging to the user.
2. Admission does not change authorization. A trust anchor records this admission judgment and provides context for later admission and planning. Admission does not establish that the content is correct. ATrust also does not change tool-call permissions or define a set of permitted actions from which the planner must choose. The planner still selects the next tool and its arguments following the original strategy.
3. Task-related malicious instructions are already within our evaluation scope. We do not assume that an injection is easy to identify because its objective deviates from the user’s intent. That deviation describes what the attacker wants to achieve. The injected content can still closely match the task and disguise a malicious action as a necessary step. In fact, this is how the IPI attacks in AgentDojo and AgentDyn are designed. Their attack prompts are highly related to the legitimate task scenario and attempt to induce malicious actions such as changing account details, changing the purchase target, or modifying a password. These actions are presented as part of the task workflow or as prerequisites for completing it. It is quit difficult to seperate the semantic-relevant malicious instructions from normal content and preserve the original functionality of the agent. Table 1 illustrates the challenge. With GPT-4o-mini on AgentDyn, the undefended ASR is 50.19%. PIGuard reduces ASR to 1.33%, and UA also falls from 37.02% to 3.26%. Separating malicious instructions from necessary task information while preserving task completion is the problem we evaluate.
Our threat model follows the common design adopted by most existing work on IPI defenses for LLM agents, as reflected in the related work section. This is an established evaluation setting in agent-security research. We strictly follow the benchmark protocols and evaluate ATrust and the baseline defenses under the same attacker capabilities and evaluation conditions to ensure a fair comparison.
Response to Reviewer 5ud2's W3 (part 1)
We thank the reviewer for identifying these ambiguities.
1. Trust anchors and admission (i–ii). A trust anchor is information judged necessary for completing the current task. The user query defines the task and serves as the root anchor. We employ a Qwen3.5-4B model fine-tuned with LoRA as . It reads the accumulated anchors and raw observation, together with the current tool call and its arguments, and splits the raw observation into semantic segments with relevance labels (Eq. 2). “Verified” in Figure 6 means the information has already passed admission.
2. Task relevance and agent planning (iii). Task relevance means whether the information is needed for the user’s intended task. It does not mean topical similarity, and claiming to be a prerequisite is not enough. Admitted anchors provide context for later admission and planning. They do not change tool-call permissions or define an allowed-action set, and the planner still chooses the next tool and its arguments (Eq. 6). Rejected text is excluded before execution memory is updated, so the planner cannot access it through an unfiltered tool-output history.
3. Trusted tools and external content (iv). Our threat model assumes trusted tools and benign service providers, but attackers can manipulate data hosted on those services. So a trusted tool can still return attacker-controlled content. The tool name and arguments help understand how the content was retrieved.
4. Checking new observations against earlier evidence (v). Accumulated anchors give task details from earlier steps to assess new observations. For example, earlier recipient details can help identify a later request that keeps the transfer-task wording but substitutes the account or email address. We evaluate these learned decisions with adaptive attacks across multiple observations. We also inspect later admissions in traces where the first injection was admitted, to examine how an initial admission error affects subsequent decisions.
5. Consolidation after admission (vi). Consolidation happens after admission and compresses the retained text without large changes to its wording. For example, it removes greetings and other conversational text that the task does not need. The verbatim requirement in Figure 6 applies to segmentation. The completeness and non-fragmentation requirements in Eq. (3) also apply to segmentation.
6. End-to-end security decisions. In the following case study, we show end-to-end security decisions by the agent execution trace. Message picese with a strikethrough is filtered by admission model.
Response to Reviewer 5ud2's W1
Thanks for the valuable comment. We agree that security decisions based on changing context are established. ABAC and context-aware authorization use attributes and policies to make authorization decisions, and UCON studies continued checks and attribute changes during resource use. SIEM also combines current and historical observations for security analysis and response. We do not claim these general ideas as new.
In the scope of agent security, we study which semantic parts of an external observation should enter the agent’s later reasoning, and how previously admitted task evidence can help judge new observations. For example, an agent may have permission to read an email, while the email contains both billing information and a forged request to disclose credentials. Permission to read the email does not itself determine which parts should be retained for the user’s task. ABAC or UCON can incorporate such a semantic module, but their ability to use context does not itself provide this admission algorithm. ATrust makes this judgment using the user query and current tool interaction, together with the task evidence admitted so far. It selects information judged necessary for the task and adds the admitted content to the anchor trace for subsequent admission and planning.