- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Speculative Directive Search (SDS), a black-box indirect prompt injection (IPI) method targeting LLM agents protected by multi-layer defenses. SDS decomposes attack payloads into four searchable components and employs a calibrated proxy to rank candidate payloads, followed by repeated verification against the target agent.
Critical strengths
-
The paper investigates prompt injection attacks against combined defenses involving input detection, prompt enhancement, and model alignment. It highlights the potential trade-off between detection evasion and malicious instruction execution, providing a meaningful setting beyond single-layer defense evaluations.
-
SDS decomposes attack payloads into four components and combines proxy-guided ranking with target-side verification to reduce ineffective search. This structured formulation also facilitates the analysis of different attack factors.
-
The evaluation covers multiple victim models, defense configurations, and two benchmarks. Component ablations, calibration sensitivity, query-budget analysis, and repeated verification experiments provide useful evidence for understanding the proposed method.
Critical weaknesses
-
SDS excludes proxy calibration queries from its online budget, while final Beta-CI confirmation requires an additional 6–16 evaluations. Moreover, Table 4 reports an average of 36.4 queries for TAP, exceeding SDS's online budget of 30. It remains unclear whether the total query costs and budget constraints are fully comparable across methods.
-
SDS trains a gradient-boosted calibration model using success labels obtained from the target defense stack and adopts leave-one-task-out calibration. Although the target task is excluded, the degree of independence between calibration and evaluation tasks remains unclear. In particular, the generalization of a proxy calibrated with only 30 samples to unseen task categories or different defense configurations requires further investigation.
-
Table 6 shows that fixing the SURFACE component to a reference form reduces ASR to zero across all four defense stacks. This suggests that attack effectiveness may depend heavily on specific payload surface forms or structured-content boundary manipulation, rather than the joint optimization of all four components. The current ablations do not fully isolate these contributions.
-
In Appendix Table 7, adding Beta-CI confirmation increases the reported ASR compared with majority-vote verification alone. For example, ASR on Stack A increases from 0.94 to 1.00. Since Beta-CI is described as a final confirmation step, this difference requires clarification regarding the evaluation protocol or experimental variability. Additionally, the main results lack corresponding uncertainty estimates.
Questions for authors
-
Could the authors report the complete target-model query costs, including proxy calibration, online search, repeated verification, and final Beta-CI confirmation, and clarify whether all baselines follow consistent budget constraints?
-
Are the calibration tasks fully independent of the final evaluation set? Could the authors further evaluate the transferability of the calibrated proxy to unseen task categories or different defense configurations?
-
Table 6 shows that fixing SURFACE eliminates attack success across all four defense stacks. Could the authors further distinguish the contribution of structured boundary manipulation from the other three components and evaluate whether the method remains effective without relying on such boundary operations?
-
Could the authors explain why adding Beta-CI confirmation increases the reported ASR in Table 7 and provide confidence intervals or repeated-run statistics for the main results to better establish their reliability?