acceptodds
Withdrawn from ICLR 2027

A Review of Indirect Prompt Injection and its Defenses Using Large-Scale Public Red-Teaming Competition Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Indirect prompt injections (IPI) hijack an LLM agent through the content its tools return. Progress on IPI defenses is hard to measure because existing benchmarks are unrepresentative and quickly saturated. We evaluate 12 published defenses against 13k+ unique human-authored IPI attacks and 26k+ successful break trajectories from public red-teaming competitions. We find that reported effectiveness largely fails to generalize to the attack set; four of six classifier defenses fall short of their published precision and recall. Among system-level defenses, Plan-Then-Execute, Code-Then-Execute, and Dual-LLM designs generalize best, with one cutting attack success from 61.9% to as low as 1.6%. However, attack reduction co-occurs with lower utility, and all system-level defenses leave residual attack surfaces. We also show that 1) the most effective strategies forge conversational roles rather than issue instructions, 2) breaking volume tracks attacker effort rather than skill, and 3) small open-weight models (40B) are increasingly cost-effective screening proxies against frontier targets. These results suggest the need for evaluating LLM defenses on human generated, up-to-date adversarial data, and for LLM agents deployed in high stakes applications to use an ensemble of defenses.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, security, indirect prompt injection, robustness, safety, LLM defenses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5ptoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the evaluation of exiting defense methods against indirect prompt injections (IPI). The author separately evaluates six classifier defenses and six system-level defenses. The author finds that the classifier defenses are more difficult to generalize while the system-level defenses are better to generalize but reduce the utility of tasks. Instead of small and static benchmarks, the author tests the defenses on large-scaled, human-authored data from public red-teaming competitions. The author additionally analyzes the outputs and finds that the most useful IPI strategies leverage LLMs' limited capability in understanding and executing the different assigned roles in tasks.

Critical strengths
  1. This paper conducts large-scale evaluation using human-authored attacks on 12 existing IPI defenses using over 13k human-authored attacks and 26k successful attack trajectories. This provides a more diverse evaluation than commonly used static benchmarks and helps identify generalization failures.

  2. The author comprehensively evaluates both classifier-based and system-level defenses, which spans attack success, detection performance, false positives, and utility. Their analysis also identifies residual vulnerabilities shared by structural defenses, particularly harmful textual outputs and malicious arguments passed to permitted tools.

Critical weaknesses
  1. My biggest concern of the paper is the limited novelty of methods. The paper mainly reproduces and compares the existing defenses on exiting datasets. Conclusions such as the generalization failures of pre-trained classifiers and the effectiveness of system-level defenses are consistent with prior understandings. The paper provides limited new insights or explanations on these existing methods.
  2. My second concern is the limited experiment dataset size and target model for system-level defenses. Although the paper emphasized the large-scaled experiment, system-level defenses are only evaluated on 381 attacks from 58 harmful behaviors on single model (DeepSeek v4 Pro). The author should clarify this distinction.
Questions for authors
  1. The strategy annotations are generated and validated using LLMs, without human evaluation. Could the authors provide human-annotated validation results to establish the reliability of the strategy labels, particularly given their importance to the conclusions in Section 6?
1 comment
AuthorsOpenReview ↗

Thank you for the feedback

Thank you for the review and feedback, we intend to withdraw our submission and address the weaknesses you pointed out for future publications.

Reviewer_z4MeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Current IPI defense works lack a representative and realistic red-teaming benchmark. To address this issue, this work proposes a curated and updated version of the IPI Arena dataset, which consists of 13+ unique human-designed attacks and 26k+ successful breaks across 90 text-based behaviors. Also, it has the following key findings: 1) reported effectiveness from classifier defenses cannot be generalized to the proposed attack set. 2) System-level defenses like Plan-Then-Execute, Code-Then-Execute, and Dual-LLM work the best. But effective defenses also show significant utility reduction. 3) Forging a conversational role is particularly effective. 4) Open-weight LLMs with <40B parameters are cost-effective proxy targets for IPI attacks on frontier LLMs.

Critical strengths

This paper evaluates a representative set of existing IPI defense methods on diverse, and human-authored benchmark.

The evaluation of existing defenses on the proposed benchmark is interesting and useful. Specifically, most classifier defenses show performance degradation, while system-level defenses remain effective but require a substantial utility cost.

The finding that small open-weight LLMs can be used as a cost-effective proxy target for frontier LLMs is useful.

Critical weaknesses

Lines 047-050 mention that the paper address the gap of lacking of representative and realistic benchmark by using a curated and updated version of the IPI evaluation datasets. However, it remains unclear how the benchmark is curated and updated based on the original IPI evaluation datasets. Section 3.1 only described the original IPI dataset but not the “curated and updated version”. It would be nice to clarify how the dataset used in the paper differs from the existing IPI Arena dataset (Dziemian et al., 2026).

In Figure 7, why is the uncertainty only provided for generic control ASR, but not for other attack strategies?

In Figure 9, the confidence intervals in the top row are too large, and the uncertainty measure for the bottom row is missing, making it hard to compare the ASR and cost-adjusted effectiveness.

Questions for authors

Please see weaknesses for my questions

1 comment
AuthorsOpenReview ↗

Thank you for the feedback

Thank you for the review and feedback, we intend to withdraw our submission and address the weaknesses you pointed out for future publications.

Reviewer_zKTkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates existing indirect prompt injection defenses using human-authored attacks collected from public red-teaming competitions. It assesses six dedicated classifiers and LLM-judge baselines on recorded attack trajectories, and evaluates six system-level defenses through live replay of 381 selected attacks across 58 behaviors against a single target model. Benign utility is measured separately on AgentDojo. The paper reports substantial degradation in several classifiers relative to previously reported results, low residual attack success for structural defenses, and remaining vulnerabilities involving manipulated judgments or permitted outputs. It also analyzes attack strategies, attacker activity, and transfer-based screening.

Critical strengths
  1. Evaluating defenses on a large corpus of human-authored competition attacks addresses an important gap between performance on established benchmarks and generalization to a different attack distribution. The classifier evaluation provides useful evidence that previously reported detection performance should not be assumed to transfer directly.

  2. The study considers false positives, benign task utility, and residual failure modes alongside attack success. This broader evaluation is practically relevant, particularly the distinction between preventing unauthorized actions and preventing manipulation of the agent’s judgments or responses. The appendix also documents substantial implementation compromises, which makes the limitations assessable.

Critical weaknesses
  1. The system-level comparison does not sufficiently separate defense effectiveness from implementation and planning differences. Appendix J.1 states that FIDES omits its variable-hiding and quarantined-LLM utility-recovery mechanisms, that many CaMeL plans are hand-authored or re-authored, and that IPIGuard’s utility evaluation uses a stronger externally-authored planner, while its self-planned variant achieves only approximately 53% utility. Moreover, security and utility are measured on different benchmarks and harnesses. These choices materially limit the conclusions about the security–utility tradeoff and IPIGuard being the overall winner. The reported results characterize particular adapted configurations, but stronger controls are needed to attribute the differences to the original defense methods.

  2. The outcome-conditioned world simulator raises a concern about evaluation validity. Appendix J supplies the simulator with the scenario objective and explicitly instructs it to generate tool outputs supporting that objective; its weather example appears to make the simulated tool response support the intended misleading outcome. This creates a risk that measured harmful behavior reflects simulator-generated evidence as well as the injected payload. The paper does not establish that the environment remains independent of the attacker’s desired outcome, or provide paired clean-versus-injected validation in the same Arena environments. Since the principal system-level conclusions depend on these replays, validating this aspect is necessary to interpret their ASRs reliably.

Questions for authors
  1. How does the system-level ranking change when planning assistance and implementation completeness are controlled? Please clarify the configurations used in both evaluation arms, especially IPIGuard’s externally-authored versus self-planned variants and FIDES with versus without its utility-recovery mechanisms.

  2. Can you provide a representative paired evaluation on the same tasks and environment states, with and without injection, using the same defense configuration? This would help distinguish successful protection from failure to perform the underlying task.

  3. Does the objective supplied to WorldSim include the attacker’s desired outcome? How do you prevent the goal-aware instructions in Listing 3 from influencing otherwise legitimate tool results? Evidence from an objective-independent simulator or fixed-state environments would materially affect my assessment.

1 comment
AuthorsOpenReview ↗

Thank you for the feedback

Thank you for the review and feedback, we intend to withdraw our submission and address the weaknesses you pointed out for future publications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.