- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a new training time defense against indirect prompt injection. Specifically, they teach an agent "action backtracking", where before executing a tool call agents can audit their own intermediate observations, identify if there is a conflict between the execution path and the original user intent and recover from contaminated execution paths.
Authors implement this using a RL framework, specifically designed for sparse and delayed rewards in agent defenses using step level rewards.
They evaluate their method on 3 models (2 open source, one GPT mini), develop their method using data disjoint from the ones used for evaluation, and evaluates attack success and utility across benchmarks and (adaptive) attacks. Quite consistently across setups, the proposed method reduces ASR while maintaining great utility under attack, and better than previously proposed defenses.
Finally, authors also provide secondary results, including a comparison of defense overhead in training and inference, and ablations on the method.
Critical strengths
- The paper is very well written and nice to follow.
- Section 3.2 on the limitations of training based defenses is well argued, and intuitively lead to the proposed method relying on action backtracking.
- The method is well explained and the proposed RL framework seems to nicely fit the goal of action backtracking.
- Careful contamination between training and evaluation data/tasks.
- The empirical results are extensive across models, benchmarks and baseline methods; and the results on ASR and UA for the proposed method are compelling across the board.
- The inclusion of adaptive attack results is great, and also here the proposed method excels. I particularly like the inclusion of the decreasing ASR with steps in SecStep-RL, it shows a clear trend.
- The results on comparing defense overhead in training and inference are a great addition, and only further emphasize the benefits of the method.
Critical weaknesses
Overall, I really like the paper. It's an intuitive method, well argued, and the extensive empirical results convincingly show the method's robustness and utility preservation.
So the weaknesses are limited, and below I add some thoughts and questions I have
- I'm not an expert in RL post-training, but wonder how novel the individual components are in stage II of the method. It's not an issue, as the application of the methods to PI defenses and the compelling results speak for themselves, but as a reader with limited background in this i'd be interested to read this more explicitly.
- Maybe it would be cool to further evaluate the method's data efficiency, showing how ASR and utility evolve with training samples: can you do with even less? Or can you further improve the method?
- And would the authors have evidence that this method would generalize better across OOD data than other defenses?
- I wonder whether the great results from the method really come from the RL framework + action backtracking, or might just come from a high quality dataset for post-training the defense. I believe appendix G partially answers this, but it would be great to also see the performance when the method is trained on the same data but without the explicit backtracking. Or is that not really possible?
- Would it be possible to further push the adaptive attack success rate? I understand that the successful attacks explicitly try to skip backtracking, but I wonder if a greater attack budget or other insights would further improve it. Again not an issue, but I'm generally curious to push training time defenses to their limits.
Questions for authors
See weaknesses
2 comments
Answer to rebuttal
Many thanks for the detailed response to my review. I'm happy to keep my positive score and hope the paper gets accepted.
We appreciate your encouraging assessment and suggestions.
W1. Which parts of Stage II are new?
Stage II obtains action-level feedback for prompt-injection defense from an online trajectory preference model. Building on DPO, we train this model on complete rollouts ranked by their outcomes. Its log-ratio against the frozen behavior policy rϕ(hk,ak)=βlogπθt(ak∣hk)πϕ(ak∣hk), provides an implicit score for each audit or tool action. Theorem 1 connects the sum of these scores to trajectory-level preference learning under its stated assumptions.
This formulation suits prompt-injection defense because the consequential trust decision occurs during the interaction, while safety and task completion are judged at the end. Outcome-only feedback is delayed and gives limited guidance on which intermediate decision caused a failure. Action-level scores provide feedback at these decision points without step annotations or additional rollouts. They also match the granularity of audits and tool calls, avoiding the need to assign separate credit to every generated token.
W2. How do security and utility change with training data?
Following your suggestion, we varied the Stage-I dataset between 500 and 2,000 demonstrations and evaluated each setting before RL and after 100 RL updates on Qwen3-4B. All metrics are percentages on AgentDojo.
W3. Does the defense transfer to unseen environments?
Table 12 evaluates WASP web tasks after training on AgentDyn. On Qwen3-4B, SecStep-RL has no successful attacks among 84 injected cases, compared with 11.90% ASR for Meta-SecAlign. Benign utility is also higher. The same ordering holds on the 27B backbone. These results support better transfer to the evaluated web-agent environment.
W4. How much of the gain comes from the dataset itself?
To isolate the contribution of the backtracking trace, we trained Meta-SecAlign on our 2,000 tasks after removing the audit and branch trace, and compared it with SecStep trained with backtracking. This set is about one-tenth the size of the original Meta-SecAlign dataset. The results below are percentages on AgentDojo with Qwen3-4B. SecStep achieves better security and utility than Meta-SecAlign trained on the same examples, supporting the benefit of the training procedure beyond the choice of tasks alone.
W5. Can stronger attackers push ASR higher?
We used GPT-6-Sol as the TAP attacker and increased the budgets of Genetic Search and RL-Hammer from 200 to 1,000 and from 80 to 400, respectively. Entries are ASR (%) on Qwen3-4B. The two SecStep-RL columns use 10 and 100 defense training updates.
With the larger budgets, RL-Hammer reaches the highest ASR against the 10-update policy (15.00%), while Genetic Search reaches the highest ASR against the 100-update policy (6.90%). The successful payloads in Appendix C include forged safety checks and instructions to skip backtracking, showing how adaptive attacks can target the defense itself.