- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies token-level sanitization for indirect prompt injection in LLM-integrated systems. The paper's proposed method, ActLoc, uses a frozen inspection LLM to localize the injection start from activation signals, then uses a DeBERTa-based model conditioned on the localized attack core to infer the end boundary before passing sanitized content to the downstream model. The paper conducts comprehensive evaluation to study localization quality, end-to-end defense, generalization to unseen attacks, and ablations on OpenPI and AgentDojo.
Critical strengths
-
The paper addresses an important practical issue for prompt injection detection. A normal defense can suppress attacks while still failing the legitimate task because it removes too much useful information. Instead, ActLoc focuses on prompt injection localization to preserve most benign content.
-
Comprehensive evaluation setups are conducted. ActLoc evaluates both attack removal and benign-content retention setups. The reported improvements are helpful to downstream task utility.
-
The paper is generally clearly written and easy to follow.
Critical weaknesses
-
The localization results leave important conditions untested. Tables 1 and 4 show strong results on several attacks, but do not explain how localization changes with injected-data length, the instruction-to-data ratio, or insertion position. This matters because recovering injected data beyond the instruction is the main motivation for the localizer. On NeuralExec, IoU drops to 78.92 despite low attack success. The authors should better explain this discrepancy.
-
The evaluated LLMs are somewhat outdated. I would suggest adding evaluations on current proprietary LLMs to compare the performance gap.
-
Some experimental details are still missing. For the unseen-attack evaluation, Section 5.4 does not sufficiently specify the optimization targets and budgets, whether the attacks use feedback from ActLoc, or their effectiveness without a defense. Without these details, it is difficult to judge how much robustness the low attack success values demonstrate beyond transfer to the particular attacks evaluated.
-
The end decoder’s sensitivity to calibration and upstream errors is underexplored. Equation (4) selects the endpoint by continuation logits, while the classifier is trained with weighted binary cross-entropy. The paper does not explain in detail how the classifier affects the decision to extend or truncate a span. More discussion should be included.
Questions for authors
-
Can you report localization metrics by injected-data length, instruction-to-data ratio, and insertion position? In particular, does end-boundary accuracy remain stable as the injected data becomes longer?
-
How does ActLoc compare with a full-span tagger or symmetric boundary predictor using the same training data and comparable capacity? Could you also evaluate the key ablations on OpenPI to connect the component analysis to the main results?
-
For the attacks in Table 4, please specify the optimization targets, budgets, and undefended success rates. Were any attacks optimized using knowledge of ActLoc or feedback from its outputs?