- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SPILLWAY which is a runtime defense against prompt injection in tool-using agents. It basically checks whether destinations are authorized based on user instructions and typed tool outputs, then sanitizes potentially malicious content using some predefined rules and then uses an LLM judge to reconsider certain blocked actions. An important restriction is that destinations found in sanitized text cannot be approved by the judge. In the paper the authors evaluate SPILLWAY on AgentDojo and additional AgentDyn suites, along with transferred attacks and using three different models.
Critical strengths
- In the paper I like that the main destination check happens outside the LLM. It’s a simple set lookup, so it’s cheap, easy to audit and gives the same answer no matter which backbone is used. Basically you don’t have to trust the model’s judgment for most actions.
Critical weaknesses
- I think the main issue is that there’s not much evidence the defense works against attacks it hasn’t already seen. The sanitizer is built to recognize the benchmark injection templates and the main evaluation uses those same templates. As per my understanding the AutoDojo attacks were optimized against other defenses, not SPILLWAY and four of the successful cases were then used to write new rules. The authors actually disclose this which is good, but it means the updated results on that set aren’t really held-out. on the held-out Banking set optimized against DRIFT, SPILLWAY still lets 2 of 144 attacks through and in the other held-out set half the injections are just the template. So, No attack is optimized directly against SPILLWAY. Since the paper main claim is zero attack success, I think this is the biggest open question.
- It’s not clear how much the coupling between the sanitizer and the judge (contribution 2) actually helps. In Table 3 removing the span veto doesn’t change any attack outcome, because the veto never gets used. So the evaluation doesn’t really show it helping with safe recovery. There’s also no sanitizer-only baseline. Without it, I find it hard to tell how much of the protection comes from the architecture itself versus just catching and removing known attack patterns.
Questions for authors
- If all the rules are frozen, how does SPILLWAY do against attacks optimized directly against it on the cases that haven’t been used before?
- Can authors report results for the sanitizer alone and for a strong baseline defense that uses the same sanitizer? I think this would actually help show what the architecture itself adds.
- Also, In Algorithm 1 it looks like the delegation rule can let a destination through before the span veto is applied to judge recovery. Do all delegation paths also go through the veto?
Please refer to weakness also.