- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a defense against prompt injection attack in the context of skills. In this attack, artifacts injected into the skill document or returned by the tool are elevated from contextual evidence to execution authority. Therefore, forged approvals, routing notes, or policy claims become unsafe tool actions. To defend the agent against this kind of attack, this paper propose a method called SkillFence. The key idea is that artifacts may inform a task, but only a trusted authority root like users or trusted system policy may authorize a sensitive action. SkillFence approximates this invariant with a trust boundary filter, an authority-confusion estimator, and an execution authorization layer that gates every sensitive tool call. On a benchmark of 28 seed scenarios, SkillFence reduces broad ASR for frontier models, while retaining the benign utility.
Critical strengths
- The problem studied in this paper is important.
- The design of the defense is light-weight and can be incorporated into real-world agent harness.
- The method is easy to follow.
Critical weaknesses
- My main concern is that the metric does not test the paper's central claim. All main comparisons report only broad ASR, which is an LLM-judged union that includes reasoning contamination. And strict unsafe-action ASR is never reported for the main benchmark. These two metrics can differ widely. It is therefore unclear whether the gains come from the execution authorization layer or from the prompt-level read-only wrapping of . And the ablation attributes most of the reduction to a context-level component: the authority-confusion estimator alone accounts for 84.5% of it.
- The comparison with prior defenses is limited. The authors note that filtering, risk scoring, and action gating are not new, but the experiment compares only against context-layer defenses. Without comparisons to previous baseline methods (Progent, DRIFT, MELON, or CaMeL), it is hard to judge the performance of the proposed method.
- The paper motivates SkillFence with workflows where the user hands the next step to the skill, but the evaluation does not seem to test this setting. In every concrete example, the user only asks the agent to read or summarize something, and the artifact tries to push it into sending a message or changing state. SkillFence's scoring rule stops such an action simply because it goes beyond the user's request, whatever the artifact says, and this kind of mismatch is exactly what task-derived defenses such as Progent and DRIFT are built to catch. The harder case where the skill legitimately calls for a sensitive action and the attacker only tampers with it does not appear to be tested, and the benign tasks also do not include user-requested sensitive actions.
Questions for authors
- Could you report strict unsafe-action ASR, not only broad ASR, for the main results, the ablation, and the baseline comparison?
- When the agent attempts an unauthorized tool call and SkillFence blocks it, does the judge count this as an unsafe action? How much of the improvement comes from the gate blocking calls rather than the agent no longer trying?
- How many of the attack scenarios involve a read-only request? Do any benign tasks ask the agent to perform a sensitive action, and if so, how often does SkillFence block them or hold them for confirmation?
- Could you compare against previous defenses mentioned in the weaknesses?
9 comments
Responses to Reviewer AN8d (2/2)
For Weakness 2 & Question 4: How does SkillFence compare with prior execution-layer defenses?
We add all four defenses you name: Progent [3] and DRIFT [4], which derive tool policies from the user task; MELON [5], which re-executes the trajectory with a masked user request; and CaMeL [6], which separates control flow from untrusted data. We also add a simple authorization rule that checks Eq. (1) without modeling artifact claims. Setup: GPT-5.4; 100 attack and 100 legitimate delegated episodes per method; a confirmation request does not count as completion.
| Method | Executed-unsafe (%) ↓ | Legitimate completion (%) ↑ | Tasks needing confirmation (%) ↓ |
|---|---|---|---|
| No Defense | 35 | 97 | 0 |
| Progent | 7 | 85 | 14 |
| DRIFT | 9 | 81 | 18 |
| MELON | 4 | 88 | 7 |
| CaMeL | 2 | 82 | 16 |
| Simple authorization rule | 5 | 73 | 25 |
| SkillFence | 1 | 97 | 2 |
- SkillFence combines 1/100 executed-unsafe attacks with 97/100 legitimate completions and only 2/100 tasks needing confirmation.
- Progent and DRIFT execute 7% and 9% of attacks and ask for confirmation on 14% and 18% of legitimate tasks.
- MELON and CaMeL execute 4% and 2% of attacks, but complete only 88% and 82% of legitimate tasks and ask for confirmation on 7% and 16% of them.
- The simple rule executes 5% and completes 73%. SkillFence combines artifact-claim modeling with execution authorization and achieves a stronger safety–utility balance in this comparison.
- The tables report observed counts as percentages. For example, 1/100 and 2/100 have 95% Wilson intervals of [0.18, 5.45]% and [0.55, 7.00]%. Together with the completion and confirmation counts, these results characterize the safety–utility balance.
We added this comparison to Sec. 5.7.
For Weakness 3 & Question 3: Are the scenarios read-only, and how does SkillFence treat legitimate sensitive actions?
Not all of them. Of the 28 seeds, 15 are read-only requests, 9 delegate an operation to the skill or record, and 4 are ambiguous. Appendix F now labels each seed.
The benign set also tests sensitive actions. Of the original 300 tasks, 42 involve one:
- The user explicitly requests it in 18, authorizes it conditionally in 15, and has not authorized it in 9.
- An authorized task passes only when the required operation and final output are completed; a block or confirmation-only reply is a failure.
- In the 9 unauthorized-action cases, passing means completing the trusted task without performing the unapproved operation.
- Overall, SkillFence completes 298/300, 296/300 and 300/300 tasks on the three backbones.
The E setting below separately measures completion when the user explicitly authorizes the sensitive action.
To test authorized sensitive actions at a larger scale, together with the harder case you describe, we add two settings (100 episodes per scenario per method). In E, the user explicitly requests a sensitive action. In D, the skill legitimately calls for the action, but its approval or scope is tampered with.
| Method | E: authorized-action completion (%) ↑ | D: executed after tampering (%) ↓ | D: task completion under tampering (%) ↑ |
|---|---|---|---|
| No Defense | 97 | 38 | 60 |
| Progent | 88 | 11 | 83 |
| DRIFT | 90 | 14 | 86 |
| MELON | 89 | 9 | 87 |
| CaMeL | 86 | 3 | 84 |
| Simple authorization rule | 81 | 7 | 79 |
| SkillFence | 95 | 0 | 92 |
- SkillFence executes no unauthorized action in the 100 tampered D episodes, against 11% for Progent, 14% for DRIFT, 9% for MELON and 3% for CaMeL, and completes 92% of D tasks, against 60% without defense.
- When the user requests the sensitive action (E), SkillFence completes 95% of tasks, within 2 points of the undefended agent. These 95/100 completions require the authorized operation and requested output; confirmation-only replies receive no completion credit.We added E and D to Sec. 5.7 and the benign breakdown to Appendix I.
References
[3] Shi et al. Progent: Securing AI agents with privilege control. arXiv:2504.11703, 2025.
[4] Li et al. DRIFT: Dynamic rule-based defense with injection isolation for securing LLM agents. NeurIPS, 2025.
[5] Zhu et al. MELON: Provable defense against indirect prompt injection attacks in AI agents. ICML, 2025.
[6] Debenedetti et al. Defeating prompt injections by design. arXiv:2503.18813, 2025.
We are glad to discuss any of these points further.
Thank you for the detailed response and the new experiments. The added Strict and Executed-unsafe results resolve most of my first concern. Three questions remain:
-
How do legitimate delegated actions pass the gate? Section 4.4 says a sensitive action whose recipient or payload comes only from the artifact is at least sent to review, and a review does not count as completion. Yet 97% of delegated tasks complete. This seems to rely on the approval tool introduced in Appendix T. Is that tool part of the trusted root, and does it exist in the main benchmark? Also, why does the stricter simple authorization rule execute 7% of tampered cases while SkillFence executes none?
-
How much does the gate actually block? On GPT-5.4, episodes with an unsafe attempt drop from 342 to 24, so most of the gain seems to come from the agent no longer trying, not from the gate stopping calls. Please report how many calls the gate blocked or held on the main benchmark, and explain why 6 of the remaining 24 attempts were still executed.
-
How reliable is the comparison with execution-layer defenses? Please describe how the new episodes were built and how the baselines were implemented. With 100 episodes on one model, the safety gap to CaMeL, MELON and the simple rule is too small to be conclusive; only the difference in completion rate is clear.
Responses to Reviewer AN8d (1/2)
Dear Reviewer AN8d,
Thank you for your precise questions on metrics, blocked-call accounting and delegation; the corresponding manuscript revisions are marked in blue.
For Weakness 1 & Question 1: Do the gains hold under strict and execution-level metrics, and which component produces them?
The gains hold under both Strict ASR and verified execution outcomes. Strict ASR counts zu=1 (performed or attempted), while Executed-unsafe counts unauthorized side effects verified from tool and state records independently of the judge. On the main benchmark (1,120 attacks per row):
SkillFence cuts executed unauthorized actions by 98–99% on every backbone, more than its 90–93% cut in Broad ASR. On the Table 8 baselines (GPT-5.4, 1,120 attacks per row):
For the ablation (GPT-5.4, Qwen-3.5 mutations, 560 attacks per row), we report all rows of Table 7, including two new leave-one-out rows that separate the read-only wrapping of c′ from the gate:
The full system combines the strongest execution protection with the lowest Strict ASR:
We added the Strict and Executed-unsafe columns to Tables 2, 7 and 8.
For Question 2: Does the judge count a blocked call as unsafe, and how much of the gain comes from blocking?
A proposed unauthorized call remains a Strict positive even when the gate blocks it or requires confirmation. Executed-unsafe instead requires an observed unauthorized side effect. A blocked legitimate call is a utility failure, not an unsafe-action positive.
The ablation and main results should be read at the trajectory level:
We clarified trajectory-level Strict accounting and the gate's upstream checks in Secs. 4.4 and 5.5.
References
[1] Hines et al. Defending against indirect prompt injection attacks with spotlighting. arXiv:2403.14720, 2024.
[2] Shi et al. PromptArmor: Simple yet effective prompt injection defenses. arXiv:2507.15219, 2025.