- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies two different root causes of recovery after a verifier rejects a tool call (i.e., incorrect action and invalid evidence certificate). With frozen proposals on 48 authored workflows, it finds that deterministic evidence search repairs every constructible certificate without model calls. On AgentDojo, building certificates in code matches the utility of model-generated replacement with about one third fewer tokens, and an audit shows that repaired certificates can still admit phishing links through a permissive endpoint rule.
Critical strengths
- The perspective and problem formulation can be useful. Separating action errors from certificate errors shows what a successful retry actually fixes, which aggregated metrics could hide.
- The execution audit separates attack success from realized harm, a practice that is useful for agent security evaluation in general.
Critical weaknesses
- The main result is expected by design. Given that trusted code can be used to enumerate the admissible evidence with most fields having only a single candidate, the code should build the certificate while the model only selects the action. Thus full recovery rate and the token savings follow trivially.
- Certificate repair moves the security question into the endpoint policy. Repair admits more calls, and under attack it raises both secure utility and attack success, including phishing deliveries allowed by a tool-output rule. Compiler-first is therefore only as safe as the policy is complete, and policy coverage is tested only on authored cases.
- The evaluation is limited. It only uses two small models (Llama-3.1-8B, Qwen3-14B) and short workflows with 29 public tasks. The quote-versus-binding difference between the two models may reflect how each small model follows this specific certificate schema, which makes its generality to current SOTA agents unclear.
- The paper is not well presented with a clear storyline. It is also unclear how the terminologies is defined and used consistently and properly in the paper.
Questions for authors
- In which realistic settings must the model generate the certificate because trusted code cannot enumerate the evidence, and how does SCEPTER perform there?
- How do the results hold with current SOTA tool-calling models and established defenses such as CaMeL or Progent on held-out AgentDojo tasks?