- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EAB, a benchmark for text-and-tool embodied agents. EAB focuses on failures that occur during task execution. These failures include tool faults, user interrupts, and state-dependent events.
EAB contains 400 cases arranged as 200 matched pairs. It uses 25 pass/fail/NA criteria. Thirteen criteria use declared anchor windows. The benchmark combines anchored grading, paired interventions, and a validated evaluation harness.
The harness separates agent execution from grading. It also uses golden samples, mutation-tested gates, deterministic replay, and a fingerprinted dataset. The authors evaluate eight LLM agents. Three agents also run in simulator, solo, and prescribed modes.
The experiments show a large difference between terminal task success and full process compliance. They also show a large performance gap between simulator and solo conditions. The paper attributes much of the simulator failure to invocation discipline.
Critical strengths
- The evaluation pipeline is carefully designed and auditable. The benchmark grades execution facts instead of only final responses. It separates execution from grading and treats NA as a separate outcome. It also uses golden samples, mutation tests, deterministic replay, and a fingerprinted dataset. These choices make evaluation errors easier to inspect and reproduce.
- The paired-intervention design is useful. The benchmark contains 183 controlled pairs. Each pair changes one declared environment variable. It also contains 17 rule-ablation pairs that serve as a null control. This design gives more localized evidence than comparisons across unrelated tasks.
Critical weaknesses
The construct validity of the main process-compliance signal is unclear. Several invocation-discipline criteria are evaluated relative to an expected execution specification. Importantly, the paper explicitly notes that an agent may accomplish the intended behavior using a different tool-call decomposition—for example, creating a route and attaching its actions in a subsequent call—yet still be penalized by P1. The worked example further shows that a single deviation from the reference decomposition can be associated with failures of multiple criteria, including action_catalog_resolve, key_args_match, no_extra_tool, skill_match, and tool_chain_match.
This raises two related concerns. First, some reported “execution failures” may reflect deviation from a hidden reference execution specification rather than genuinely incorrect or undesirable behavior. Second, correlated criteria can cause one underlying deviation to contribute multiple fail verdicts, which complicates the interpretation of the claim that 61.4% of failures are due to invocation discipline. Because invocation discipline is central to the paper’s empirical conclusions, I view this as a substantive construct-validity issue rather than a presentation detail.
Questions for authors
- For a representative sample of P1 failures, how many executions are functionally correct but differ only from the reference tool-call decomposition? A manual audit or an equivalence-aware re-scoring would materially affect my assessment.
- The paper states that one off-reference call can fail several P1 criteria. How does the reported 61.4% invocation-discipline share change when one underlying deviation is counted once per episode?
3 comments
Thank you for the detailed response and additional analyses. I appreciate the blinded manual audit and the episode-level recount, which help clarify the interpretation of the P1 results and address my original concerns. These revisions are helpful, and my overall assessment remains positive. I will maintain my original rating and have no further questions.
We thank the reviewer for the follow-up and for the constructive questions. The audit and the episode-level recount have made the paper clearer about what the P1 family measures, and both will be kept in the final version (Sec. 3.5, Sec. 4.4, App. L). We appreciate the reviewer's positive assessment and the time spent on our submission.
We thank the reviewer for the careful reading, and for recognizing that the evaluation pipeline is carefully designed and auditable: run–grade separation, NA as its own outcome, golden samples, mutation tests, deterministic replay, and a fingerprinted dataset are there precisely so that errors in the evaluation itself can be inspected and reproduced. The revision adds App. K, which describes the four layers of the harness and how all domain facts are isolated in a domain pack, separate from the evaluator code; we invite the reviewer to take a look. Below we answer both questions with recounts from the released traces and a blinded manual audit, and we thank the reviewer for working through the App. H trajectory criterion by criterion. Changes are marked in blue in the revised PDF.
What P1 measures. P1 scores conformance to the execution contract of the deployed system, not only whether the end state is reached; terminal success is reported on a separate channel precisely so that the two can disagree (App. F.2). The part of this contract that matters most here is the speech rule: the gateway's production system prompt states that the final text reply is converted to speech automatically and that no extra tool may be called for speech, so an
audio_speakcall makes the robot say the same thing twice. The revision states this in Sec. 3.5 and prints the rule in App. D.4.Q1. The reference agent has 1,026 P1-failing simulator episodes. 75.3% contain an
audio_speakcall; 19.8% fail only on arguments or ordering, with no extra call; 11.7% are the create-then-update decomposition the reviewer highlights; 8.7% contain other unrequested advancing calls (pause, resume, etc.).Manual audit (App. L). We drew a stratified sample of 100 P1-failing episodes across these categories with a fixed seed. Two annotators labeled each episode independently, without communicating during annotation, from the dialogue, call arguments, tool receipts, and final state; they did not see grader verdicts, categories, or the reference chain. Labels: (A) functionally correct, differing only in decomposition; (B) no functional error, but an unrequested advancing action; (C) a functional error; (D) a case or simulator defect. Raw agreement was 94% (Cohen's κ=0.87). Weighted to the population, type A is 5.8% (95% CI [0.0, 12.3]) for one annotator and 12.9% ([4.0, 21.8]) for the other; type B is 32.4% and 28.2%; type C is 61.8% and 58.9%; no episode was labeled D. By human judgment, then, at least 87% of P1-failing episodes contain a functional error or an unrequested advancing action. In the create-then-update category specifically, the annotators labeled 1 and 3 of the 30 sampled episodes as type A; all others were labeled B or C.
Q2. Sec. 4.4 notes that the shares count verdicts, not independent root causes. Recounted per episode over the reference agent's 1,200 simulator episodes, 1,098 of which have at least one failure: 93.4% of failing episodes contain an invocation-discipline failure, and these fail on average 3.0 of the five criteria. Counting invocation discipline once per episode, its share is 34.6%; counting each of the three failure classes once per episode, it is 42.9% (terminal shortfall 28.6%, other 28.6%). Under every counting it is the largest class. F2 rests on family-level per-case rates (P1 14.5% vs. P3 72.1%), which count each case once however many criteria one deviation trips. The revision reports the episode-level share in Sec. 4.4 and Fig. 5.
Revisions. (1) Sec. 3.5 states that P1 measures conformance to the deployed execution contract, and App. D.4 prints the speech rule. (2) The new App. L reports the audit. (3) Sec. 4.4 and Fig. 5 report the episode-level share. (4) App. H separates the causes of each verdict in the trajectory, replacing the statement that the failure is pure invocation discipline. (5) The recount script, audit sample, and labels are in the supplementary material.
We hope these analyses address the construct-validity concern. The reviewer noted that a manual audit would materially affect the assessment: the blinded audit finds that at least 87% of P1-failing episodes contain a functional error or an unrequested advancing action, and the revision makes the measured contract explicit. If our response resolves the reviewer's concerns, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.