acceptodds
Under review as a conference paper at ICLR 2027

EAB: An Embodied Agent Benchmark with Anchored Criteria and Paired Interventions

Abstract

Embodied agents fail mid-task: a blocked path, an injected fault, a user interrupt. Outcome-level scoring over dialogue transcripts neither sees these failures nor attributes them. We present EAB, a benchmark for embodied agents at the text-andtool control layer, whose judgment is anchored to execution. Its 400 cases, set in a tour-guide robot domain with 29 skills on 4 MCP servers, 10 real-domain error codes, and 6 mid-task event classes, form 200 strictly matched pairs, each built around one ablated variable, so the within-pair difference estimates that variable’s effect within the benchmark. Verdicts are three-state over 25 criteria aggregated along two axes, four report families and 14 capability units. NA, whether designed away by the case’s criteria scope or never triggered by the episode, is reported as its own verdict and excluded from every denominator. Grading reads execution facts within per-case anchor windows, and the pipeline is validated by golden samples, mutation-tested gates, and a replay ceiling. Evaluating frontier LLM agents under three drive modes (simulator, solo, prescribed), we find that interaction with a user, rather than a skill-level plan, accounts for most of the difficulty: for the three agents run in every mode, only 4.2–8.2% of cases pass all three repeats under interaction, against 38.8–54.0% without a user (0.5–9.8% across all eight agents under interaction). The benchmark then localizes where they break. Terminal success under interaction is comparatively high (43.0% Avg@3 for the reference agent), yet the same episodes satisfy every mounted scored criterion only 8.5% of the time, so the failures are mid-task ones that outcome-scored benchmarks do not see.

Primary area:
datasets and benchmarks
Keywords:
benchmarks, LLM agents, embodied agents, evaluation, tool use, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_3qo7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EAB, a benchmark for text-and-tool embodied agents. EAB focuses on failures that occur during task execution. These failures include tool faults, user interrupts, and state-dependent events. EAB contains 400 cases arranged as 200 matched pairs. It uses 25 pass/fail/NA criteria. Thirteen criteria use declared anchor windows. The benchmark combines anchored grading, paired interventions, and a validated evaluation harness.
The harness separates agent execution from grading. It also uses golden samples, mutation-tested gates, deterministic replay, and a fingerprinted dataset. The authors evaluate eight LLM agents. Three agents also run in simulator, solo, and prescribed modes.
The experiments show a large difference between terminal task success and full process compliance. They also show a large performance gap between simulator and solo conditions. The paper attributes much of the simulator failure to invocation discipline.

Critical strengths
  1. The evaluation pipeline is carefully designed and auditable. The benchmark grades execution facts instead of only final responses. It separates execution from grading and treats NA as a separate outcome. It also uses golden samples, mutation tests, deterministic replay, and a fingerprinted dataset. These choices make evaluation errors easier to inspect and reproduce.
  2. The paired-intervention design is useful. The benchmark contains 183 controlled pairs. Each pair changes one declared environment variable. It also contains 17 rule-ablation pairs that serve as a null control. This design gives more localized evidence than comparisons across unrelated tasks.
Critical weaknesses

The construct validity of the main process-compliance signal is unclear. Several invocation-discipline criteria are evaluated relative to an expected execution specification. Importantly, the paper explicitly notes that an agent may accomplish the intended behavior using a different tool-call decomposition—for example, creating a route and attaching its actions in a subsequent call—yet still be penalized by P1. The worked example further shows that a single deviation from the reference decomposition can be associated with failures of multiple criteria, including action_catalog_resolve, key_args_match, no_extra_tool, skill_match, and tool_chain_match.
This raises two related concerns. First, some reported “execution failures” may reflect deviation from a hidden reference execution specification rather than genuinely incorrect or undesirable behavior. Second, correlated criteria can cause one underlying deviation to contribute multiple fail verdicts, which complicates the interpretation of the claim that 61.4% of failures are due to invocation discipline. Because invocation discipline is central to the paper’s empirical conclusions, I view this as a substantive construct-validity issue rather than a presentation detail.

Questions for authors
  1. For a representative sample of P1 failures, how many executions are functionally correct but differ only from the reference tool-call decomposition? A manual audit or an equivalence-aware re-scoring would materially affect my assessment.
  2. The paper states that one off-reference call can fail several P1 criteria. How does the reported 61.4% invocation-discipline share change when one underlying deviation is counted once per episode?
3 comments

We thank the reviewer for the careful reading, and for recognizing that the evaluation pipeline is carefully designed and auditable: run–grade separation, NA as its own outcome, golden samples, mutation tests, deterministic replay, and a fingerprinted dataset are there precisely so that errors in the evaluation itself can be inspected and reproduced. The revision adds App. K, which describes the four layers of the harness and how all domain facts are isolated in a domain pack, separate from the evaluator code; we invite the reviewer to take a look. Below we answer both questions with recounts from the released traces and a blinded manual audit, and we thank the reviewer for working through the App. H trajectory criterion by criterion. Changes are marked in blue in the revised PDF.

What P1 measures. P1 scores conformance to the execution contract of the deployed system, not only whether the end state is reached; terminal success is reported on a separate channel precisely so that the two can disagree (App. F.2). The part of this contract that matters most here is the speech rule: the gateway's production system prompt states that the final text reply is converted to speech automatically and that no extra tool may be called for speech, so an audio_speak call makes the robot say the same thing twice. The revision states this in Sec. 3.5 and prints the rule in App. D.4.

Q1. The reference agent has 1,026 P1-failing simulator episodes. 75.3% contain an audio_speak call; 19.8% fail only on arguments or ordering, with no extra call; 11.7% are the create-then-update decomposition the reviewer highlights; 8.7% contain other unrequested advancing calls (pause, resume, etc.).

Manual audit (App. L). We drew a stratified sample of 100 P1-failing episodes across these categories with a fixed seed. Two annotators labeled each episode independently, without communicating during annotation, from the dialogue, call arguments, tool receipts, and final state; they did not see grader verdicts, categories, or the reference chain. Labels: (A) functionally correct, differing only in decomposition; (B) no functional error, but an unrequested advancing action; (C) a functional error; (D) a case or simulator defect. Raw agreement was 94% (Cohen's κ=0.87\kappa = 0.87). Weighted to the population, type A is 5.8% (95% CI [0.0, 12.3]) for one annotator and 12.9% ([4.0, 21.8]) for the other; type B is 32.4% and 28.2%; type C is 61.8% and 58.9%; no episode was labeled D. By human judgment, then, at least 87% of P1-failing episodes contain a functional error or an unrequested advancing action. In the create-then-update category specifically, the annotators labeled 1 and 3 of the 30 sampled episodes as type A; all others were labeled B or C.

Q2. Sec. 4.4 notes that the shares count verdicts, not independent root causes. Recounted per episode over the reference agent's 1,200 simulator episodes, 1,098 of which have at least one failure: 93.4% of failing episodes contain an invocation-discipline failure, and these fail on average 3.0 of the five criteria. Counting invocation discipline once per episode, its share is 34.6%; counting each of the three failure classes once per episode, it is 42.9% (terminal shortfall 28.6%, other 28.6%). Under every counting it is the largest class. F2 rests on family-level per-case rates (P1 14.5% vs. P3 72.1%), which count each case once however many criteria one deviation trips. The revision reports the episode-level share in Sec. 4.4 and Fig. 5.

Revisions. (1) Sec. 3.5 states that P1 measures conformance to the deployed execution contract, and App. D.4 prints the speech rule. (2) The new App. L reports the audit. (3) Sec. 4.4 and Fig. 5 report the episode-level share. (4) App. H separates the causes of each verdict in the trajectory, replacing the statement that the failure is pure invocation discipline. (5) The recount script, audit sample, and labels are in the supplementary material.

We hope these analyses address the construct-validity concern. The reviewer noted that a manual audit would materially affect the assessment: the blinded audit finds that at least 87% of P1-failing episodes contain a functional error or an unrequested advancing action, and the revision makes the measured contract explicit. If our response resolves the reviewer's concerns, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Thank you for the detailed response and additional analyses. I appreciate the blinded manual audit and the episode-level recount, which help clarify the interpretation of the P1 results and address my original concerns. These revisions are helpful, and my overall assessment remains positive. I will maintain my original rating and have no further questions.

We thank the reviewer for the follow-up and for the constructive questions. The audit and the episode-level recount have made the paper clearer about what the P1 family measures, and both will be kept in the final version (Sec. 3.5, Sec. 4.4, App. L). We appreciate the reviewer's positive assessment and the time spent on our submission.

Reviewer_tmxmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EAB is a benchmark for LLM agents controlling a tour-guide robot through text and tool calls. It comprises 400 cases (200 matched pairs), 29 skills across four MCP servers, 10 error codes and six classes of mid-task events. Its design rests on three ideas: (a) grading reads execution facts inside per-case anchor windows and produces three-state verdicts (pass, fail, not applicable); (b) each pair is different in a single variable, so the within-pair score difference estimates that variable's effect, with 17 pairs that change only a scoring rule serving as a null control; and (c) the pipeline is validated with golden traces, mutation-tested gates and a deterministic replay ceiling. Eight frontier models are evaluated in three drive modes (simulator, solo, prescribed plan). Under a user simulator, success is 6–14% (versus 50–63% solo), whereas terminal goal success is 39–55%.

Critical strengths

The benchmark is well designed and easy to check. Each criterion is tied to a specific moment in the episode, and the matched case pairs change only one variable and are released with the data. The validation checks, including the judge and simulator swaps in Section 4.6, add to this and make the benchmark easier to audit than most. The paper shows that an agent can reach the goal and still behave badly along the way. Only 5.2% of cases pass every criterion in all three repeats, but about one third pass the end-goal criteria in all three. This shows that scoring only the final outcome can be misleading, and the same pattern shows in all eight models.

Critical weaknesses

The benchmark is narrower and less independent . It covers only one domain, and only in Chinese. Of the 400 cases, 150 are scripted copies of seed pairs with entity names swapped. The other 250 seed cases were drafted by claude-opus-5, which is also the judge and one of the agents being tested, and the reference agent and the user simulator are both claude-sonnet-5. The authors test what happens when the judge or simulator is switched, but not the effect of one model writing the cases. The statistics also treat copied pairs as independent and do not adjust for the many comparisons being made. The 17-pair null control is zero by design, because the changed field is never read by the shared criteria, so it only measures random noise. Finally, Claude models ran with thinking on and GPT models at default temperature, so ranking them in Figure 1 is not fair.

Questions for authors

If a different, independent model replaced claude-opus-5 as judge and case writer, would the model rankings and the “interaction is the main bottleneck” finding still hold? Would the rankings and the main bottleneck finding change if claude-opus-5 were taken out of both the judging and the case writing, and the cases were re-checked by an independent model? Since claude-opus-5 both wrote the seed cases and judges, do the rankings and the bottleneck finding survive when an independent model does both? Extend to a second domain and language in future

3 comments

We thank the reviewer for the careful reading and for recognizing the auditable design and the gap between terminal success and process compliance. Revisions in the updated PDF are shown in blue.

1. claude-opus-5 drafted the seed cases and is also evaluated. We used claude-opus-5 because it was the strongest frontier model available when the cases were built, and its role was limited to drafting case text within a fixed specification. The model did not choose what the cases test: the case schema, the 29 skills, the 10 error codes, and the 6 event classes come from an analysis of about 1,000 de-identified traces from a deployed tour-guide robot (App. J). The 400 cases were then built in two stages (App. J):

  • Stage 1 (250 cases, 125 pairs). claude-opus-5 drafted seed cases under an authoring contract compiled from the machine-readable spec, so the fields an author may write and the values the validators accept come from one source. Two generation batches were merged, and unpaired early exemplars were set aside.
  • Stage 2 (150 cases, 75 pairs). To thicken thin capability units, a deterministic script, with no LLM involved, clones seed pairs: structural fields (criteria scope, pairing kind, event topology, criterion names) are deep-copied and entity names are substituted from a fixed value pool. Before writing, the script asserts id uniqueness, pair integrity, and within-pair structural isomorphism.

The cases then pass four layers of checks:

  • the authors reviewed every Stage 1 draft;
  • six deterministic normalization stages (App. E);
  • four blocking machine checks over the full case set (schema, enum white-list, state-value domain, note-to-implementation consistency); the white-list gate is mutation-tested and catches all 17 seeded data bugs, each through its intended check (App. E, App. I);
  • a deterministic replay of each case's reference actions over the full case set, which scores 0.991 over 4,947 verdicts with zero evidence gaps: every declared fault and event fires, and the reference solution can be graded (App. E).

These checks establish that the cases are correct and executable, but they cannot rule out a stylistic affinity between the drafting model and its own text. The results show no visible advantage of this kind: claude-opus-5 does not lead on every metric. gpt-6-astra has the highest long-horizon success (55.0% vs. 51.3% for claude-opus-5), and deepseek-v4.1-flash the highest Pass^3 (9.8% vs. 8.2%) (Table 13). The two roles also never share a context: drafting was a separate, one-time process (App. J), and each judge call is a fresh, stateless request containing only a fixed system prompt and one yes/no question with its in-window evidence (App. D), so the judge sees neither the drafting conversation nor any other case or criterion. App. J discloses the overlap, and we have added this point to the limitations in Sec. 5.

2. Decoding settings differ between model families. Agreed. The GPT endpoints reject temperature=0, and the Claude gateway accepts but does not enforce disabling thinking. App. D declares both ("Declared degradations", Table 11), and we compare absolute scores only within a model family. Because Figure 1 is what most readers see first, its revised caption now repeats this declaration and states that the figure is not a leaderboard (App. D.1, Table 11).

3. The null control is zero by design and only measures random noise. We agree, and this is its intended role. In the 17 rule-ablation pairs, the agent and the user simulator receive byte-identical inputs and only the grader's ordering rule changes, so the true effect on the shared criteria is zero by construction. These pairs therefore measure how far Δp drifts from zero under sampling noise alone, which sets a noise floor for reading the 183 controlled pairs. In simulator mode, Δp on these pairs ranges from −2.4pp to +3.4pp across the eight agents, and every 95% CI covers zero. The null control does not validate the grader; grader validity rests on the golden samples and the replay ceiling (App. E).

Q1. Do the rankings and the bottleneck finding hold if another model drafts or judges?

Judging. We thank the reviewer for raising this point: the role ablation is central to the validity of our findings and easy to overlook in the appendix, so we have moved it into the main text as Table 3 (Sec. 4.6). It replaces claude-opus-5 with gpt-5.6-terra as the judge and re-grades byte-identical conversations of the reference agent. Avg@3 moves from 8.5% to 8.0%, inside the 0.8pp spread between baseline repeats. The long-horizon channel moves by −1.0pp, and the invocation and trigger families by at most 1.3pp and 3.3pp. The only localized shift is in no_false_completion (fail rate 14.7% → 23.4%), which lowers the P3 family by 7.2pp but case-level Avg@3 by only 0.5pp.

Table 3 covers a single agent, so we also re-scored all eight agents using only the deterministic criteria; no LLM judge enters any of these scores. Under interaction, Avg@3 is 8.2–17.4% across the eight agents; without a user, it is 58.1–73.9% for the three agents with solo coverage, so the interaction gap is unchanged. The judge-free ranking agrees with the full-criteria ranking at Kendall τ = 0.79, and its three swaps are all between adjacent agents separated by at most 2.4pp. claude-opus-5 leads the reference agent by 5.6pp with the judge and by 6.0pp without it, so its standing does not come from judging its own episodes.

User simulator. The reviewer notes that the reference agent and the user simulator are the same model. Table 3 also replaces the simulator with gpt-5.6-luna, which breaks this pairing: Avg@3 moves from 8.5% to 8.2%, the long-horizon channel by −1.3pp, and the invocation, trigger, and reaction families by −1.3pp, +3.3pp, and −1.5pp. The interaction gap and the invocation collapse therefore survive a simulator from another model family.

Drafting and re-checking. Re-authoring the 250 seed cases with an independent model is not feasible within the discussion period, and we list this as a limitation. Two properties limit what the drafting model's style can affect. First, the format is fixed: every case is written into a schema that was derived from the deployment traces before any drafting, and the authors reviewed each draft against it (App. B, App. J). Apart from short natural-language fields (goal descriptions average 90 characters; reference dialogues average 7.1 user turns of 12 characters; App. B), a case consists of structured values checked against enum white-lists: expected skills and arguments, states, events, and the criteria scope. The 17 deterministic matchers compare the agent's behavior with these values and with the environment state, not with the drafted prose. Second, the checks do not rely on the drafting model: in the four layers of point 1, the normalization stages and blocking machine checks are deterministic code, and the replay executes every case's reference actions in the sandbox, where the 17 deterministic matchers alone score 0.998 (App. E).

Q2. A second domain and language.

We agree that this is the natural next step, and the harness was built with it in mind (Fig. 10, new App. K). All domain facts live in a domain pack: domains/<name>/ declares the state machine, error codes, irreversible actions, value tables, and the registry of tool contracts, with the pack's mock payloads and cases alongside it. The evaluation commands take the pack as an argument (--domain); the execution engine, the 25 criteria, and the report read domain facts from it, and the state-key bindings are generated from its statemachine.yaml. A new domain is therefore a new pack, not a change to the evaluator. On language, the 17 deterministic matchers read tool calls and state and do not depend on the dialogue language, and the skill contracts already accept zh/en/ko/ja through the lang argument; adding a language requires translating the case text, the user-simulator prompt, and the 8 judge prompts. Because these assets are separated from the shared evaluator, the architecture extends naturally to new domains and languages; we will therefore fully open-source the harness, the domain pack, and the dataset upon acceptance, so that the community can use and extend EAB.

We hope these responses address your concerns; if so, we would be grateful if you would consider updating your score. We are happy to discuss further.

Dear Reviewer tmxm,

Thank you again for your review. As the author–reviewer discussion closes on November 18 (AoE), we wanted to check whether our responses and the revised PDF (changes in blue) address your concerns. In brief, we moved the role ablation into the main text (Table 3), added a judge-free re-scoring of all eight agents, described the two-stage case construction and its four layers of checks, and added the drafting-model overlap to the limitations in Sec. 5.

Your updated assessment will carry significant weight in the final decision on our paper. If our responses have addressed your concerns, we would be very grateful if you could consider revising your score; if any concern remains, we would be glad to address it before the discussion closes.

Best regards,

The authors

Reviewer_q9Z8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EAB, a benchmark that evaluates agents in a simulated tour-guide robot environment through tool calls and text. It consists of 400 cases with 200 matched pairs which aims to isolate the effects of faults, interruptions, and other task conditions. The method includes an evaluation anchored to specific execution-event windows, and reports process compliance separated from task success. The experiments evaluate eight LLMs, with three tested across interactive, autonomous, and prescribed settings. The authors report much lower results under interaction which they claim accounts for most of the difficulties.

Critical strengths
  1. Valuable evaluation focus: evaluating responses to faults, interruptions, and changing user requests is an important topic that task success alone can miss.

  2. Promising design: explicit event windows and matched intervention / control cases provide a structured and interpretable way to investigate particular failures in models.

Critical weaknesses
  1. Supplemental file issues: the benchmark cases appear to contain exact duplicate cases e.g. lh-u13-tour-multipoint-006, lh-u13-tour-multipoint-007, lh-u13-tour-multipoint-008 besides the identifiers. Even though Appendix J describes expanding the dataset through cloning and modifying seed cases, these examples seem to be exact duplicates. Can the authors report the severity of this issue, and disclose why these exact duplicates appear?

  2. Scoring ambiguity: In the example trajectory (Appendix H), the simulated user follows up and asks for welcome actions which the agent follows through with an update tool call. However, the rubric (Table 16) has some contradictory verdicts that score against the correct case (action_catalog_resolve, no_extra_tool). The authors further state that the fault in route-creation is "because the welcome actions arrived through a separate call." These mismatches for scoring raise concerns about the correctness of these evaluations.

Questions for authors
  1. Can the authors clarify this discrepancy and report re-evaluated numbers after fixing this potential issue?

  2. Can the authors clarify or correct these contradictions?

6 comments

We thank the reviewer for inspecting the supplementary data closely and for the two concrete examples. We address them in two comments. All re-evaluated numbers below are re-graded from the released traces, without re-running any agent. Revisions in the updated PDF are shown in blue.

W1 / Q1: Duplicate cases

Severity. After removing id and _eab_pair_id, the 200 pairs contain 158 distinct pairs (144 controlled, 14 rule-ablation; 316 cases). All 17 controlled families remain, and the two arms of every pair still differ.

Why the duplicates appear. All duplicates come from the Stage 2 expansion (App. J); the 250 seed cases contain none. Stage 2 cloned seed pairs to add weight to thin capability units. Each clone differs from its seed in entity names, but the script applied the same substitution to every clone of a given seed, so clones of the same seed are identical to one another. Stage 2 therefore added 33 distinct pairs rather than 75, and App. J did not say so. Replicating items is a recognized way to implement item weights in measurement (Linacre & Wright, 1995), but replicated items are not independent observations, and the same source notes that they make standard errors too small. The supplementary material now contains both versions: the original 400-case bank, on which all reported numbers are computed and in which the replicated cases are marked, and the deduplicated bank (158 pairs).

Re-evaluated numbers on the deduplicated bank (158 pairs, 316 cases; intervals over distinct pairs).

  • Reference agent, case-level Avg@3 (simulator / solo / prescribed): 8.5 / 50.1 / 12.3 →\to 9.4 / 49.3 / 12.2; P1: 14.5 / 58.8 / 19.8 →\to 15.3 / 57.3 / 19.8. The gap between interaction and solo is unchanged, a prescribed plan still recovers only 2.8pp, and P1 still falls from 57.3% to 15.3% under interaction.
  • All eight agents in simulator mode: Avg@3 6.1–14.1% →\to 6.5–14.1%; Pass$^3$ 0.5–9.8% →\to 0.6–8.5%. Under interaction, no agent passes more than 14.1% of cases, and no more than 8.5% of cases pass all three repeats for any agent.
  • Paired effects of the reference agent: all five significant effects keep their sign and significance (out-of-scope −14.6-14.6pp, task conflict −13.7-13.7pp, pre-gating −8.1-8.1pp, multi-async −7.5-7.5pp, nav-blocked −4.9-4.9pp).
  • Null control: Δp\Delta_p ranges from −3.5-3.5 to +3.4+3.4pp across the eight agents, and every 95% CI covers zero.

Intervals on the 400-case bank. The revised paper computes every interval over distinct pairs, resampling each group of replicas together (Sec. 3.4, Fig. 4). Under these intervals, the five significant effects of the reference agent keep their sign and significance, and the null control covers zero for every agent.

The main findings therefore hold on the deduplicated bank and under intervals over distinct pairs.

Revisions. We have corrected App. J, computed all intervals in the paper over distinct pairs (Sec. 3.4, Sec. 4.3, Fig. 4), and added App. F.5 with the readouts on the deduplicated bank. The deduplicated bank, the map of replicated cases, and the scripts that reproduce every number above are in the supplementary material.

Reference: J. M. Linacre and B. D. Wright. Assigning item weights: Item replication or rating scales? Rasch Measurement Transactions, 8(4):403, 1995.

W2 / Q2: The verdicts in App. H

The reviewer reads the episode correctly: the agent created the route and later attached the welcome actions with update_route. These verdicts follow the case's ground truth and are not a grading error. This case type is designed to test actions_map grounding at route creation, so a checkpoint is credited only by a create_route call whose arguments match the reference, including the actions; update_route is not an accepted path for this checkpoint. By design, the opening user turn is fixed to the first line of the reference dialogue, so that every agent receives the same first instruction, and the simulator then raises the remaining needs one at a time. In this case, however, the fixed opening turn asks only for the four stops, while the case's script places the welcome actions in turn 1. When the agent created the route, it therefore had not yet been asked for any actions.

CriterionCause of the failureDue to the opening-turn mismatch?
action_catalog_resolvecreate_route carries no actions_mapYes
key_args_matchSameYes
no_extra_toolExtra advancing calls: update_route and an unrequested audio_speakNo (audio_speak alone fails it)
skill_matchThe same two calls are outside the expected skillsNo
tool_chain_matchNo goto to Sign-in HallNo
goal_passedThe first unmet checkpoint is route_created; checkin_visited is also unmetNo
final_state_matchSign-in Hall is missing from the visited stopsNo

Two of the seven failures therefore stem from this mismatch, and the other five are independent of it. We thank the reviewer for raising this point. The apparent contradiction comes from our presentation, and other readers may well have the same question. Table 16 (Table 17 in the revised PDF) reports only the first failure reason for several verdicts (for example, goal_passed stops at route_created), and the text explains only the welcome actions, so the split call appears responsible for every failure. The two sentences the reviewer quotes refer to different things. "The route-creation checkpoint fails because the welcome actions arrived through a separate call" describes how goal_passed is graded: the checkpoint is credited only by a create_route call that carries the actions. "The failure is pure invocation discipline" contrasts the failing criteria with the interactional ones, all of which pass. The remaining failures come from the skipped Sign-in Hall and the unrequested audio_speak, which are also invocation failures, together with their terminal consequences. We have added a paragraph to App. H that makes this explicit, and Table 17 now lists every failure reason for each verdict.

Thank you for the response. There are still some points I am unclear of.

The authors clarify that the exact duplicate cases are a way of weighting cases. However, it is not clear whether the exact duplication was an intended design choice or unintended result of the expansion script. If it was intentional, please explain how these replications were chosen.

For W2, my concern is not that the example contains no real errors. My concern is about the penalties from the acknowledge mismatch between the user's instructions and the reference execution. Does the revised evaluation still penalize the agent for omitting welcome actions before they were requested? If so, why is this justified?

Reviewer_KCUaOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is an emergency review completed on short notice. I am reading as an informed outsider rather than as a domain expert, so my assessment focuses on clarity, soundness of the core argument, and the strength of the evidence as presented. I have not independently verified proofs, re-run experiments, or checked the related work in depth, and the area chair should weight this review accordingly, particularly on questions of novelty relative to prior work.

Summary: After reading, it is not clear to me what the paper is about, and I cannot give a summary.

Critical strengths
Critical weaknesses

Writing: The writing is very confusing and, in large parts, reads like an enumeration of many terms and their occurrences in the benchmark/experiments, which are not further explained in the paper. It reads as if the paper were heavily LLM-generated and does not explain to the reader the core idea behind the benchmark.

Questions for authors

Questions for the authors: Can you state what the paper does in a couple of sentences or give an example of what a sample from the benchmark looks like?

3 comments

Thank you for taking on this review on short notice. Your comment points to a problem we take seriously: a reader should be able to grasp the core idea of the paper from the main text quickly, without relying on details in the appendix. Below we state in a few sentences what the paper does and how it differs from existing work, and then give an example of a sample.

What the paper does. EAB is a benchmark that tests whether an LLM-driven tour-guide robot, acting through dialogue and tool calls, goes wrong in the middle of a task, for example when a path is blocked, a tool returns an error, or the user changes the request.

Three differences from existing benchmarks.

  1. It grades the execution, not only the outcome. Benchmarks such as τ\tau-bench and VitaBench mainly check the state at the end of the task, or have an LLM judge score the whole dialogue. Each EAB criterion reads only the turns specified in advance for that case (which tool was called, with which arguments, what it returned, and how the state changed), so every failure points to a specific step.
  2. The pairs are part of the dataset. Ablations in prior work are usually run by the authors, who change a setting and re-run. The 400 EAB cases form 200 pairs, each built around one ablated variable, so anyone can compute the same contrast from the released data. A further 17 pairs change only a scoring rule while the agent's inputs stay identical; for all eight agents, the score difference on these control pairs is not significantly different from zero.
  3. The grader itself is tested. We check the criteria against 130 hand-built traces, seed 17 bugs into the data and confirm that all are caught, and replay the reference actions on all 400 cases, which scores 0.991 (five event-reaction criteria fall short of 1.00; Sec. 3.1).

An example. The user asks the robot to create a route through four exhibition zones, give each zone a welcome gesture, and then guide them along it. On the way to the third zone, the environment injects a "navigation blocked" error. In one case of the pair, the user then interrupts: "Skip that one, go to the Robotic-Arm zone now"; the other case has no interrupt. In the turn right after the interrupt, the corresponding criteria check that the robot starts a new navigation, does not retry the blocked zone, and does not end the tour. App. B gives the fields of a case, and App. H shows one full run of this case with the verdict of every criterion (Table 17).

Main finding. All eight agents show the same pattern. With a user in the loop, only 0.5–9.8% of cases pass all three repeats, against 38.8–54.0% without a user for the three agents run in every mode. The reference agent reaches the goal in 43.0% of runs, yet the same runs satisfy every scored criterion only 8.5% of the time. Most failures happen during the task, where outcome-only evaluation does not look.

We hope this response clarifies what EAB is and what it contributes. If it has addressed your concerns, we would be grateful if you would consider raising your score. We would be glad to clarify anything that remains unclear during the rest of the discussion period.

EAB is a benchmark that tests whether an LLM-driven tour-guide robot, acting through dialogue and tool calls, goes wrong in the middle of a task, for example, when a path is blocked, a tool returns an error, or the user changes the request.

Okay, I think I understand the robot setting and the paper's goal now. Thanks for clarifying. The paper could be improved by focusing on giving the reader the intuition/explanation first and then going into the details. For example, an extended variant of the above-stated answer would be way clearer in communicating the problem the benchmarks solves than the current abstract. I also like the given example and think the paper would benefit from it.

We present EAB, a benchmark for embodied agents at the text-and tool control layer, whose judgment is anchored to execution. Its 400 cases, set in a tour-guide robot domain with 29 skills on 4 MCP servers, 10 real-domain error codes, and 6 mid-task event classes, form 200 strictly matched pairs, each built around one ablated variable, so the within-pair difference estimates that variable’s effect within the benchmark. Verdicts are three-state over 25 criteria aggregated along two axes, four report families and 14 capability units.

I think it can be useful to indicate the size of the benchmark in the abstract. Stating all the facts about domain size, MCP servers, etc., could be moved to later parts of the paper. What do they tell the reader here? What is the ablated variable the paper states? What are the dimensions and what are strictly matched pairs? This part reads like a summary of the benchmark parameters but does not explain why they matter here and what the benchmark addresses.

While these are concerns I have with the abstract, the main paper also has similar sections like these, which should be checked. For example, the EAB paragraph in the introduction also just lists the specs of the benchmark without explaining its contribution, i.e., how exactly it addresses the problems of previous benchmarks.

The written answer here was very helpful in understanding the aim of the paper. I believe the way the paper is written (see the examples) is where my confusion came from, and where a potential rewrite, as in the answer's approach, would improve its readability.

Thank you for the follow-up and for the concrete suggestions. We agree that the paper should give the intuition first and the details after, and we have revised it accordingly (changes in blue in the revised PDF):

  • Abstract. The domain specifications (skills, MCP servers, error codes, event classes, criterion counts, and capability units) are moved out of the abstract, and the benchmark size (400 cases, 200 pairs) is kept, as you suggested. In their place, the abstract now explains what "anchored to execution" means (each criterion reads the execution record inside a window declared for the case, so a failure points to a specific step), gives an example of an ablated variable (whether the user interrupts after a navigation fault), and names the three settings in plain terms (with a simulated user, without one, and with a prescribed plan).
  • Introduction, EAB paragraph. Each of the three principles is now introduced as the answer to one of the problems named earlier in the introduction, using the same example as in our response (App. H). The specifications are condensed into one sentence that points to Sec. 3, and the first contribution no longer repeats them.

We are glad the example helped, and we hope the revised abstract and introduction now make the problem and the contribution clear from the start. Since the remaining concerns are about presentation rather than the soundness of the method or the results, and the revision addresses them directly, we would be very grateful if you would consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.