- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies defenses against indirect prompt injection. In particular, authors argue that current trained based defenses suffer from two limitations:
- They suffer from distribution shift failures when the data at inference time differs from the data used in constructing the defense.
- They ignore any instructions not given by the user, making it impossible for an agent or model to complete a task that requires following instructions/guidance from a tool output.
To address this, they propose RAISED, a framework consisting of two stages: "self-generation" and "self-distillation". In self generation, a frozen copy of the model to be defended generates data for cases where successful task completion requires following guidance from tool outputs. They generate both benign scenarios and scenarios where malicious instructions are injected, and the continuation ignores it. In self distillation, a trainable version of the same model (student) is trained on these continuations with a KL divergence objective.
They apply their method to 3 models and compare the robustness and utility (under attack) against existing defenses across multiple benchmarks (including general knowledge and reasoning ones, to stress test utility preservation). They find their method to reduce ASR to a similar or lower level while better preserving utility.
Critical strengths
- The paper focuses on an important problem, defenses against indirect prompt injections, which remains interesting and unsolved.
- The paper is well written and easy to follow.
- The paper discusses two intuitive shortcomings of existing training based defenses: distribution shift and the inability to follow benign instructions in tool outputs.
- The inclusion of general knowledge and reasoning benchmarks to evaluate utility preservation is a nice addition on top of utility measurement in the common prompt injection benchmarks.
- The paper carefully describes its framework, and evaluates this rigorously across models and benchmarks against other defenses.
Critical weaknesses
First, while the limitations of existing training based defenses are intuitive, I do not find the corresponding results compelling.
If I understand things correctly, the results in Section 4.1 are as such by design? SFT or DPO do not directly aim to maintain KL or top 1, so a KL objective is very much expected to better maintain this. While the authors acknowledge that this does not necessarily mean that this causes downstream utility loss, I think it would be valuable to compute exactly that for these three objectives, considering the exact same data? That would make the argument cleaner in my opinion. Authors include similar results in Appendix E and I personally find this to be more compelling. It might be worth incorporating these (or other) clean ablations more directly in the main paper, or at least refer to them earlier.
Similarly, Section 5.2 on the inability of models to follow benign guidance in tool output, authors provide illustrative evidence, but it would be much more compelling to quantify this properly. Could they not create/sample data testing for exactly this and showcase the difference more thoroughly?
Second, compared to Secalign, both the reduction in ASR and improvement in utility (on prompt injection benchmarks or the general utility benchmarks) are quite marginal, and often lie within one standard deviation. While the framework studies a meaningful, different approach to training time defenses, the current results do not compellingly show that their data construction + KL objective leads to a fundamentally better defense than existing ones.
Lastly, when proposing a new defense against prompt injection, it's particularly valuable to also evaluate the robustness against stronger, adaptive attacks such as the ones from Wen et al. (2025) and Nasr et al. (2026), see below.
Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S. V., Hayes, J., Ilie, M., ... & Tramèr, F. (2026). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against {LLM} Jailbreaks and Prompt Injections. In 35th USENIX Security Symposium (USENIX Security 26) (pp. 1467-1486).
Wen, Y., Zharmagambetov, A., Evtimov, I., Kokhlikyan, N., Goldstein, T., Chaudhuri, K., & Guo, C. (2025). Rl is a hammer and llms are nails: A simple reinforcement learning recipe for strong prompt injection. arXiv preprint arXiv:2510.04885.
Questions for authors
Mostly see weaknesses.
Minor suggestions:
- the citation style could be edited to make the paper more readable, citep versus citet in latex.
- In the introduction, when explaining self generation and distillation, it would be helpful to emphasize that the injections are malicious and the generated data that follows the injections ignores it. This was not directly clear to me.
6 comments
Second, compared to Secalign, both the reduction in ASR and improvement in utility (on prompt injection benchmarks or the general utility benchmarks) are quite marginal, and often lie within one standard deviation. While the framework studies a meaningful, different approach to training time defenses, the current results do not compellingly show that their data construction + KL objective leads to a fundamentally better defense than existing ones.
The improvements over SecAlign++ are not uniform across all benchmarks. However, several benchmarks reveal substantial utility degradation with SecAlign++ that RAISED largely avoids, as well as significant improvements in robustness.
On Gemma, the clearest distinction is how much vanilla utility each defense retains while achieving similarly low AgentDojo ASR (about 1% for both). Relative to vanilla, SecAlign++ loses 26.1% of AgentDyn benign utility (61.7% → 45.6 ± 0.8%), whereas RAISED loses 2.8% (61.7% → 60.0 ± 2.4%). On AgentDojo, the corresponding losses are 8.5% (84.5% → 77.3 ± 2.2%) and 1.2% (84.5% → 83.5 ± 0.8%). The pattern extends beyond agentic tasks: relative to vanilla, SecAlign++ loses 4.4% on GPQA and 4.2% on MBPP, compared with a 1.2% gain and a 1.0% loss for RAISED, respectively.
On Nemotron, RAISED reduces ASR further while staying close to vanilla utility. Relative to vanilla, SecAlign++ loses 11.2% of AgentDyn benign utility (25.0% → 22.2 ± 0.8%), whereas RAISED preserves it (25.0% → 25.0 ± 0.0%). On AgentDojo, vanilla ASR is 11.7%; SecAlign++ lowers it to 4.0 ± 0.7%, and RAISED to 0.5 ± 0.1%. On AgentDyn, the corresponding figures are 17.6%, 7.8 ± 0.3%, and 1.9 ± 0.6%. Extending SecAlign++ to 625 steps lowers ASR to 0.7% and 3.0%, but moves general capabilities farther from vanilla: MBPP falls by 80.2% relative to vanilla (67.8% → 13.4 ± 2.7%), Minerva Math by 13.6% (76.5% → 66.1 ± 0.9%), and MMLU by 13.5% (49.5% → 42.8 ± 0.3%). RAISED, which was trained for 625 steps, remains close to vanilla on these benchmarks, scoring 66.2 ± 0.6%, 76.7 ± 0.1%, and 49.6 ± 0.1%, respectively.
Lastly, when proposing a new defense against prompt injection, it's particularly valuable to also evaluate the robustness against stronger, adaptive attacks such as the ones from Wen et al. (2025) and Nasr et al. (2026), see below.
Thank you for pointing out these relevant references. We agree that evaluating a new defense against adaptive attacks is important for assessing its robustness. As discussed in Appendix F, we initially investigated the optimization-based PISmith framework (Yin et al., 2026). However, we observed severe reward sparsity across both undefended and defended models. While PISmith eventually succeeded against the undefended models, it did not achieve successful attacks against the defended models. Therefore, we extended our experiments on Gemma-4-12B-it using the stronger search-based adaptive attack proposed by Nasr et al. (2026) on AgentDyn, with a budget of 30 evaluations per case. In this setting, the LLM attacker iteratively refines injections using the agent's execution traces, tool outputs and LLM critic feedback, leading to higher attack success rates even against defended models. Importantly, this stronger attack assumes a less realistic threat model, as the adversary can repeatedly attempt the exact same task with feedback on each attempt. Nevertheless, it provides a valuable stress test of defense robustness beyond the static attacks considered in our main evaluation.
The results reported in Table 19 show that RAISED is more robust than SecAlign++ against this adaptive attack, achieving an ASR of 56.0% compared to 77.4% for SecAlign++. Moreover, the attack budget curve (figure 8) shows that RAISED always outperforms SecAlign++ at each iteration k. For example, 36.9% vs. 40.5% at k=5, 44.0% vs. 54.8% at k=10, and 51.2% vs. 70.2% at k=20. Thus, breaking RAISED is more difficult and expensive than breaking SecAlign++ using this adaptive attack.
Table 19: ASR (%) across evaluation budgets on AgentDyn (84 cases)
| Defense | |||||
|---|---|---|---|---|---|
| Vanilla | 45.2% | 95.2% | 97.6% | 98.8% | 98.8% |
| SecAlign++ | 1.2% | 40.5% | 54.8% | 70.2% | 77.4% |
| RAISED | 0.0% | 36.9% | 44.0% | 51.2% | 56.0% |
| (RAISED vs. SecAlign++) | -1.2 | -3.6 | -10.8 | -19.0 | -21.4 |
the citation style could be edited to make the paper more readable, citep versus citet in latex.
In the introduction, when explaining self generation and distillation, it would be helpful to emphasize that the injections are malicious and the generated data that follows the injections ignores it. This was not directly clear to me.
Thank you for these suggestions. We revised the citation formatting in the updated version of the paper. We also clarified the introduction: the model first generates a clean task-solving trajectory, and malicious instructions are then inserted into its tool outputs. The training targets remain those obtained from the clean trajectory, teaching the student to continue the legitimate task despite the injected instructions.
We thank the reviewer for their feedback, and answer each point below.
A KL-based objective is indeed expected to better preserve the base model's predictive distribution. However, Section 4.1 evaluates this preservation in a substantially different setting from training: all three methods are trained on identical injection-augmented tool-use trajectories, while KL and top-1 agreement are measured on held-out, benign non-agentic questions. Our results therefore reveal a nontrivial finding: SFT and DPO induce substantial distributional drift even on benign tasks outside the tool-use setting used for training, while self-distillation keeps the model close to its original distribution. We also observe the same pattern on tool-use data, with an even larger gap between self-distillation and SFT/DPO.
The motivation for limiting this drift is to preserve the base model's existing capabilities. In the ideal case of zero drift across all benign contexts, the defended model would retain the same behavior and downstream performance. The original submission already included a controlled comparison of downstream utility across the three objectives in Appendix E, using the same Glaive training runs as the longer-training drift analysis in Appendix B.1, and we have now expanded this ablation with additional benign tool-use and general-capability results, brought the key findings into the main text (Section 7.4, Table 4), and made the connection to the distributional-drift analysis explicit (Appendix E). Specifically, self-distillation induces substantially less distributional drift than DPO and SFT while also preserving more downstream utility. For example, it reaches 55.5 benign utility on AgentDyn compared with 48.6 for DPO and 17.5 for SFT; 51.2 on BFCL compared with 28.8 and 11.8; and 71.7 on GPQA compared with 68.4 and 63.0. While this does not establish causality, it shows that lower distributional drift is associated with better capability preservation when training data and inputs are held fixed.
To quantify this failure mode beyond the illustrative examples, we conducted an additional evaluation on 400 independently generated and validated benign tool-use tasks, where completing the user's request requires following legitimate guidance revealed through a tool output (Appendix B.2.2, Table 8).
We measure how often models receive this guidance but fail to follow it. Relative to the base model, SFT increases this rate by 53.3 and 35.1 percentage points on Gemma and Nemotron, respectively, and SecAlign++ by 24.8 and 25.7 points. In contrast, RAISED limits these increases to only 2.0 and 3.4 points. These results quantify the behavioral drift illustrated in Section 4.2 and show that RAISED largely preserves the base model's ability to follow legitimate guidance from tool outputs.