- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PACT, a provenance-aware permission framework for defending LLM agents against indirect prompt injection. PACT checks tool calls against task-specific contracts covering operations, arguments, destinations, data sources, and approval requirements. When a proposed action is rejected, the framework attempts recovery through a permitted alternative subject to the same checks. The paper evaluates PACT across controlled attacks, multiple models, and public agent benchmarks, and investigates the challenges of automatically constructing task contracts.
Critical strengths
-
A clear and principled execution-time defense framework. PACT separates tool-call authorization from the agent's proposed actions. Combining task contracts, provenance tracking, and checked recovery provides a well-motivated approach to prompt injection defense.
-
Extensive evaluation and valuable open-source artifacts. The paper covers multiple models, attack settings, and public benchmarks, while also acknowledging the limitations of automatically generated contracts. The released implementation, experimental records, and validation scripts are particularly valuable for reproducibility and independent examination.
Critical weaknesses
-
Dependence on High-Quality Task Contracts. PACT's strongest results rely on complete task contracts and permitted alternatives supplied in advance. When contracts are generated automatically, many argument constraints are missing or incorrectly specified, and the resulting security improvements are modest. Although the authors acknowledge this challenge, the practical effectiveness of PACT remains dependent on obtaining sufficiently accurate task contracts.
-
Further Clarification on Recovery Enforcement. Algorithm 1 requires replacement actions to pass the same safety checks before execution. However, an examination of the released code and traces suggests that some recovered actions may be recorded as successful despite failing the final persisted_state_safety check. While this does not necessarily indicate unauthorized data disclosure, it raises questions about whether the experimental recovery procedure fully implements the stated safety requirements.
Questions for authors
-
Practical Contract Construction. Could the authors elaborate on how accurate task contracts and permitted alternatives could be obtained in realistic deployments without access to reference tool calls? This would help clarify the applicability of PACT beyond the supplied-contract setting.
-
Recovery Safety Checks. Could the authors clarify why recovered actions with failed final persisted_state_safety checks can still be recorded as successful? How does the implementation ensure that replacement actions satisfy all required checks before execution?
Thank you for the detailed response and additional experiments. However, they largely confirm my main concerns: generated contracts still yield non-significant gains on the full evaluation (p = 0.0703), recovery without supplied alternatives remains weak, and under matched conditions PACT and Progent have identical decisions, safety, and utility. The contribution beyond existing permission checks therefore remains unclear, and I will keep my score.