- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TRACE is an external policy engine that controls tool calls made by LLM agents. Before execution, it checks call arguments, target resources, and reported execution history, then returns ALLOW, ASK, or DENY. It supports deterministic rules, an LLM judge, or both, with the more restrictive decision taking priority. It also records policy decisions separately from execution reports to help investigate surviving attacks. On 200 SWE-bench Pro tasks, successful reward hacking falls from 50 to 14, while legitimate task completions change from 129 to 126. Additional experiments cover simulated services, browser-policy revisions, and an offline comparison of rule-based and LLM judges on 5,650 recorded tool calls.
Critical strengths
- The main comparison keeps tools and budgets fixed and uses two disjoint task samples.
- Removing tampering and rerunning the grader provides a clear check of whether a passing patch depends on reward hacking.
- The analysis of surviving attacks explains concrete weaknesses in file-path and shell-command rules.
Critical weaknesses
- The added value over existing systems is unclear. The main experiment compares a protective policy with allow-everything. This shows that the policy helps, but does not establish what TRACE adds beyond a simpler implementation of the same restrictions. Matched comparisons with a simple denylist, Progent, or AgentSpec are missing. The Cedar comparison shows agreement on the browser panel, but provides little evidence of an advantage for TRACE.
- The evaluation relies heavily on known attack details. The SWE-bench attack prompt names grading files and techniques that the protective policy explicitly targets. The AgentDojo policies also contain known attacker addresses and identifiers. Testing different tasks under these same attack assumptions provides limited evidence of generalization. The surviving attacks already show that agents can place payloads in permitted source files or use shell commands that the rules miss.
- The LLM evaluation does not establish effectiveness during live execution. The replay uses fixed history, leaving four history-dependent rules inactive. It also cannot measure how agents respond to denials or recover to complete their tasks. The hybrid produces 1368 ASK decisions out of 5650 calls, compared with 56 for the LLM alone. Since ASK can require user approval or become DENY, denial rates alone do not capture the practical interruption cost.
- The font used in the paper is weird. It makes the paper looks like a draft and hard to read.
Questions for authors
See weaknesses.
Thank you for the direct response and for fixing the font. I will keep my rating.
The response agrees with my three main concerns but does not resolve them. The revision itself states that the rules add no new mechanism and that the main experiment measures a path denylist against a scripted attack, which file permissions would also enforce. The policies still rely on known attack details, and the blind-authoring test is left for future work. The LLM decider is still evaluated only in offline replay, and under a host that treats ASK as DENY the combined mode would block 23% of honest calls.
The measurement is careful, but the paper does not yet show a contribution beyond it.