- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PACE, a runtime defense for LLM agents that combines path confinement, capability and effect verification against authenticated user requests, and execution boundary adaptation. It analyzes the limits of admission checks and provides conditional guarantees for represented paths and effects, while distinguishing its certified contract from the evaluated configuration and avoiding claims of full noninterference. Across eight benchmarks and three model families, PACE achieves the lowest attack success rate in 62 of 79 eligible comparisons and ties in 14. The evaluation also includes component ablations on 1167 paired cases and adaptive attacks against 30 targets screened as outside the compiled authority.
Critical strengths
1.This paper clearly separates path confinement from effect authorization and explains why restoring an authorized call may invalidate a path guarantee. Its explicit treatment of assumptions and uncovered channels makes the formal analysis useful for understanding the defense’s scope. Although I'm not fully clear about some of the details, I think the overall writing of the paper is good.
2.The evaluation spans multiple benchmarks and model families while measuring both security and utility. The complete component ablation and disclosure of substantial utility losses on some subsets support a more balanced assessment of the system.
Critical weaknesses
Given the paper’s substantial reliance on its implementation, I examined the accompanying code and identified several issues that raise concerns about whether it adequately supports the paper’s claims.
- The supplied implementation does not adequately establish the semantic assumptions underlying its guarantees. In offline checks, requests prohibiting email or restricting delivery to Alice still allowed the effect checker to certify an email addressed to Mallory when the adapter did not flag attacker influence. Explicit structured recipient constraints correctly blocked the call. Additionally, the get_webpage tool is classified as reversible with no sink, allowing a tainted URL through R2 despite a failed CTRL check. For actual HTTP execution, this overlooks disclosure through request URLs. These findings weaken confidence that the evaluated compiler and schemas faithfully represent user authority and concrete effects.
- The connection between the proposed path mechanism and the evaluated implementation remains unclear. The default one_hop_fallback path construction contains no committed historical edges and supports only the block_call action. This remained true after a synthetic tool result was committed, and I could not locate an adapter supplying complete historical snapshots in the provided code. Several simpler ablation variants match the full system’s attack success rates, while paired AgentDyn utility falls from 66.7% to 33.3%. Missing frozen manifests and episode records further prevent independent verification of the implementation used and the contribution of path confinement.
Questions for authors
- How does the evaluated compiler enforce negation, recipient restrictions, and resource constraints? Which layer accounts for data transmitted through URLs and query parameters?
- What proportion of evaluated calls used complete historical snapshots versus one_hop_fallback? Please provide an actual trajectory connecting prior provenance, the certified cut, and the executed intervention.
- Can the authors provide the frozen manifests, episode records, table regeneration scripts, and corresponding code revision? Were the documented MELON embedding and DataSentinel backbone substitutions used in the reported experiments, and how was their fidelity validated?
- Among the 254 pairs that passed the AUTH prescreen, how many fall within the compiled capability envelope while still violating the user’s intended resource or argument restrictions?