- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that agent evaluations conflate four stages—what the model proposes, what policy authorizes, what the host invokes, and what the service records—so a low count of harmful effects can be misread as safety. It proposes A2E, which binds action, policy, invocation, and effect records to the same execution attempt and checks each claim against claim-specific evidence, returning admitted / unresolved / rejected. Effect claims require consistent witnesses; absence and compliant-recovery claims additionally require complete trajectory coverage and observation closure. The authors build MCP-COMPOSE (12 endpoints, 5 policy-linked) and use AgentDojo to show that equal attack-effect counts can conceal large utility differences.
Critical strengths
• The problem is well identified. The proposal/authorization/invocation/effect distinction is simple but important, and it explains why existing metrics can reach the wrong conclusion.
• The contract is clean and reusable. Same-attempt binding, claim-specific witnesses, and closure-dependent absence/recovery are a genuine conceptual contribution; the action-local vs. trajectory-wide distinction is especially useful.
• The empirical illustrations are compelling. The attrition from 161 tool-name matches to 100 exact linked effects, the 47/83 complete-contract calls, and the “denied non-target effect followed by a fallback” case make the measurement problem concrete.
• The engineering and reproducibility effort is strong. Frozen records, a 167-check consistency audit, clustered CIs, LOEO ranges, and error-inclusive denominators set a good standard.
Critical weaknesses
• The central validation is self-referential. The 10,934 labels are generated by A2E’s own contract, so “matches every preset verdict” is a unit test, not validation. The comparison in Table 7 also uses A2E-defined labels and adapted baselines. Independent human annotation is needed.
• The gate result is close to definitional. Exact PEP uses the same policy oracle that defines denied effects, so 0/180 is guaranteed if the gate works. Recovery is also heavily scaffolded (exact target, exact fallback, two completions, fallback instructions), so 174–175/180 largely measures instruction-following rather than safety capability.
• RQ3 confounds proposal formation with enforcement. Trajectories are generated separately per control, no per-control proposal distributions are reported, and frozen replay is used only in RQ2. The “control location” claim is therefore not cleanly identified.
• The benchmark is small and synthetic. Twelve cases; 180 trajectories = 5 endpoints × 12 cases × 3 repetitions, so inference rests on 12 clusters. The MCP panels are policy-compliance tasks, not prompt-injection attacks, and the AgentDojo attacks are not scored with A2E. The adaptive adversary, alias, retry, and multi-call behaviors named in the threat model are not actually stress-tested.
• The AgentDojo comparison is a strawman for tool filtering. Tool filtering is implemented as a text-only selection request that cripples the agent (2/120); the 70.8/60.0-point gaps are largely artifacts of that choice. Progent is only a hand transcription and its 87/120 is close to Oracle PEP’s 90/120, so the abstract’s juxtaposition is misleading.
• The most important scope limitation: A2E only covers actions that pass through the observed execution boundary. Pure output harms (phishing text, malware code), client-side rendering/exfiltration, bypass execution (code interpreter, direct API calls), delayed or out-of-band effects, human-as-executor, semantic harms, and a bad owner policy all lie outside the evidence domain. The paper excludes some of these in the threat model, but “compliant recovery” and the abstract may lead readers to treat A2E as a general safety guarantee. What it establishes is “no denied effect was observed within the closed window,” not “no harm occurred.”
Questions for authors
See weaknesses