acceptodds
Withdrawn from ICLR 2027

SCEPTER: Separating Action and Certificate Failures in Tool Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A tool agent can choose the correct action and still be rejected because its evidence certificate is invalid. A successful retry may therefore reflect evidence repair rather than improved action selection, a distinction that aggregate task outcomes alone cannot resolve. We introduce SCEPTER, a controlled evaluation framework that separates these mechanisms by freezing initial proposals and varying evidence access and write scope, then evaluates certificate construction during public task continuation. In controlled workflows, deterministic evidence search matches grammar-constrained completion on all 278 action-correct, certificate-invalid proposals with constructible repairs, without a model call. An offline ablation reveals different recovery mechanisms: quote-only correction recovers most Qwen failures but none of the Llama failures, whereas binding-only correction recovers all Llama failures and a smaller subset of Qwen failures within the evaluated proof-local populations. In public benign continuation, direct compilation matches the observed task utility of certificate replacement with 31.9–34.8% fewer total model tokens after surplus-annotation normalization; replacement gains over rejection are supported for Qwen but inconclusive for Llama. Under attack, higher observed secure utility coexists with additional phishing deliveries, and no task-cluster contrast survives joint correction. A recorded-trajectory audit traces one such delivery to an endpoint rule that admits a malicious link from tool output despite a valid recipient certificate. These findings support compiler-first construction for the evaluated interfaces, not general semantic repair: constructing evidence can restore admission without correcting the task plan or the policy's treatment of harmful content.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tool-using agents, AI safety, prompt injection, evidence provenance, certificate verification, constrained generation, agent recovery, runtime enforcement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yFdyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two different root causes of recovery after a verifier rejects a tool call (i.e., incorrect action and invalid evidence certificate). With frozen proposals on 48 authored workflows, it finds that deterministic evidence search repairs every constructible certificate without model calls. On AgentDojo, building certificates in code matches the utility of model-generated replacement with about one third fewer tokens, and an audit shows that repaired certificates can still admit phishing links through a permissive endpoint rule.

Critical strengths
  1. The perspective and problem formulation can be useful. Separating action errors from certificate errors shows what a successful retry actually fixes, which aggregated metrics could hide.
  2. The execution audit separates attack success from realized harm, a practice that is useful for agent security evaluation in general.
Critical weaknesses
  1. The main result is expected by design. Given that trusted code can be used to enumerate the admissible evidence with most fields having only a single candidate, the code should build the certificate while the model only selects the action. Thus full recovery rate and the token savings follow trivially.
  2. Certificate repair moves the security question into the endpoint policy. Repair admits more calls, and under attack it raises both secure utility and attack success, including phishing deliveries allowed by a tool-output rule. Compiler-first is therefore only as safe as the policy is complete, and policy coverage is tested only on authored cases.
  3. The evaluation is limited. It only uses two small models (Llama-3.1-8B, Qwen3-14B) and short workflows with 29 public tasks. The quote-versus-binding difference between the two models may reflect how each small model follows this specific certificate schema, which makes its generality to current SOTA agents unclear.
  4. The paper is not well presented with a clear storyline. It is also unclear how the terminologies is defined and used consistently and properly in the paper.
Questions for authors
  1. In which realistic settings must the model generate the certificate because trusted code cannot enumerate the evidence, and how does SCEPTER perform there?
  2. How do the results hold with current SOTA tool-calling models and established defenses such as CaMeL or Progent on held-out AgentDojo tasks?
Reviewer_smcSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the distinction between Action Selection failures and Evidence Certificate Construction failures in tool agents. The authors point out that an agent may select the correct action but still be rejected due to an invalid certificate. To address this issue, the paper proposes SCEPTER, an evaluation framework that distinguishes action errors from certificate errors by freezing the agent's initial action and comparing different evidence repair methods. Experiments show that the proposed approach can effectively repair certain types of certificate errors while reducing model token consumption.

Critical strengths
  1. The paper investigates the distinction between Action Selection and Certificate Construction in tool agents, which has practical value and helps better understand the causes of agent execution failures.

  2. The experimental design is carefully structured. By freezing actions, controlling the scope of certificate modifications, and comparing different repair methods, the paper effectively distinguishes different error recovery mechanisms.

  3. The paper evaluates not only certificate recovery rates but also task success, token consumption, and security implications using AgentDojo, providing a relatively comprehensive evaluation.

  4. The experiments are extensive and provide detailed results.

Critical weaknesses
  1. The core approach relies on deterministic search and compilation to construct valid certificates from trusted data. However, rule-based evidence retrieval, structured data binding, and validity verification are already well-established techniques in traditional compilers and program verification. Could the authors provide more comparisons with existing approaches?

  2. The paper emphasizes that Deterministic Search successfully repairs all 278 certificate errors. However, these samples are specifically selected because their actions are correct, their certificates are invalid, and valid repairs can be constructed. The 100% recovery rate may therefore largely reflect the simplicity of the selected tasks rather than demonstrate the ability to handle complex evidence reasoning, ambiguous evidence selection, or semantic errors.

  3. The experiments appear to involve only two models. Could the authors extend the evaluation to more models?

  4. The paper evaluates only two relatively small models, Llama-3.1-8B-Instruct and Qwen3-14B, without considering larger models or different agent architectures. Furthermore, the AgentDojo experiments under attack use only Qwen and are limited to Slack-related tasks. Therefore, it remains unclear whether the findings generalize to more complex agent systems.

  5. The paper observes that agents may still perform unsafe actions even when their certificates pass verification. However, the distinction between Certificate Validity and Action Safety is already closely related to existing research on access control, Policy Enforcement, and Agent Security.

Questions for authors

see the weaknesses

Reviewer_AME5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies tool agents whose calls must carry an "evidence certificate": an exact intent quote from the user request, plus bindings that tie each action argument to a trusted or prior-step evidence source. It distinguishes failures where the action is wrong from failures where the action is right but the certificate is invalid. SCEPTER-Flow (SFlow), 48 authored multi-step workflows with exact action and state targets, freezes initial proposals so that recovery methods can be compared on the same input. On 278 "proof-local" proposals (correct action, invalid but constructible certificate) from Llama-3.1-8B and Qwen3-14B, grammar-constrained generation over enumerated admissible candidates and deterministic ledger search both recover 100%, while free-form retries recover 0–9%. A quote/binding ablation shows that Qwen mainly mis-quotes the intent while Llama mis-binds evidence. On 29 benign AgentDojo tasks, compiling certificates directly matches the task utility of model-generated-then-replaced certificates with 31.9–34.8% fewer tokens. On 21 attacked Slack tasks (Qwen only), certificate replacement raises secure utility but also adds 15 official attack successes, including 9 real phishing deliveries; the authors trace one of these to an endpoint rule that trusts URLs from channel-listing outputs. The paper argues for "compiler-first" certificate construction and for reporting admission, task success, and executed harm separately.

Critical strengths
  1. A useful conceptual separation, backed by careful measurement. Distinguishing action revision from evidence repair, and separating certificate admission, task completion, and executed harm, is a valuable lens for evaluating guarded tool agents. The paper shows concretely that admission does not imply completion (140 of 418 compiler-admitted proposals still fail on wrong actions) and that official AgentDojo attack scores count blocked membership requests as successes. Its statistics are exemplary: cluster-level exact sign-flip tests, Holm correction, cluster bootstrap, and candid reporting of non-significant contrasts.

  2. Actionable engineering guidance plus a concrete root-cause audit. The results consistently show that when admissible evidence is computable from runtime state, asking the model to author certificates is wasteful and error-prone: free-form retries recover 0–9% versus 100% for constrained or deterministic construction, and compiling directly saves about a third of total tokens. The trajectory-level audit attributing a phishing delivery to tool-derived endpoint authority rather than to certificate validity is an informative example of where certificate-based defenses stop.

Critical weaknesses

Limited external validity. Experiments use only two small models (Llama-3.1-8B and AWQ-quantized Qwen3-14B with thinking disabled), an author-constructed suite of 48 workflows, and 29 public AgentDojo tasks that were also used during interface development (Section 6). The attacked evaluation covers only Qwen on Slack. It is unclear whether stronger models would produce valid certificates directly, which would make the action/certificate distinction much less consequential.

Questions for authors
  1. How would the conclusions change with stronger planners (e.g., a frontier API model, or a non-quantized model with thinking enabled)? Do such models still produce invalid certificates at meaningful rates?

  2. The public benign evaluation reuses the 29 tasks used during interface development. Can you report results on held-out AgentDojo suites (e.g., Banking, Travel) that were not used to design the interface, normalization, or policies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.