acceptodds
Under review as a conference paper at ICLR 2027

CRM-Agent-Bench: Beyond Response-Only Evaluation for Tool-Using Enterprise Agents

Abstract

Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. Full-suite strict success therefore measures satisfaction of heterogeneous task-declared contracts rather than universal verification of every state dimension. Grading the complete deployed agent this way asks whether it satisfies the declared process and state requirements, where applicable, and at what scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Tool-Using Agents, Agent Evaluation, Stateful Benchmarks, Executable Contracts, Multi-Turn Interaction, Enterprise AI, CRM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EG67OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper introduces CRM-Agent-Bench, a synthetic, headless benchmark for enterprise tool-using agents, containing 555 tasks across nine domains and 122 tools. Tasks define executable contracts over required actions, arguments, ordering, semantic responses, target states, and prohibited actions/arguments/states. Strict success requires all declared checks to pass; cost and latency are reported separately. Sixteen systems are evaluated using a shared router/specialist scaffold and fixed user-simulator and judge configurations. The paper finds lower success under complete contracts than under response-only grading and provides component-level diagnostics and qualitative examples of execution failures despite plausible replies.

Critical strengths

The paper separating semantic response checks from actions, arguments, ordering, and declared state effects prevents a plausible answer from overriding an execution failure.

All 16 systems receive valid grades on the same 555 tasks, with no infrastructure exclusions in the reported runs. The paper discloses state/negative-check incidence, the single-trial protocol, and the distinction between final evaluations and legacy qualitative traces. These details improve interpretability.

Critical weaknesses

The added value over existing benchmarks is not attractive. ToolSandbox already checks tool-use trajectories, AppWorld checks final state and unintended changes, and EnterpriseOps-Gym covers stateful enterprise workflows. The paper cites these works, but needs to show more clearly what its new tasks or contract design reveal beyond them. Although the authors outline some subtle differences from their perspective in Table 1, I still feel the research questions and findings overlap significantly with prior work.

Most models in Figure 3 achieve similar scores, with Claude Sonnet 5, GPT-5.5, and GPT-5.6 Sol scoring exactly the same. This raises concerns about the benchmark’s ability to distinguish model capabilities.

The evaluator needs stronger validation. Human agreement is not reported for the final response judge, and the full scoring rules need testing on valid alternatives and unintended state changes. Figure 4 also mixes tasks with and without response checks, making the reported gap difficult to interpret.

Questions for authors
  1. How well does the final judge agree with human annotators? Please also test the full evaluator on valid alternatives, duplicate writes, and unintended state changes.

  2. Do the three models tied at 451/555 fail on the same tasks? Are the small ranking differences stable across repeated runs?

  3. Please report Figure 4’s response/strict pass–fail table on the same 402 tasks. How much of the gap comes from the other 153, and what failures explain the remaining gap?

10 comments
AuthorsOpenReview ↗

Author Response to Reviewer EG67: Mechanism Comparison, Multi-Run, Human Reliability Studies, and Matched 402-Task Decomposition

We thank Reviewer EG67 for the constructive review and insightful questions. We have updated the manuscript with a mechanism-level benchmark comparison (Appendix L), a stratified human agreement study with three independent human annotators (Appendix I), deterministic replay probes and a manual business-execution audit of the programmatic evaluator (Appendix J.3 and Appendix K), three-round stability analyses (Appendix B), and a fully decomposed response-vs-strict contingency table on the matched 402-task set (Figure 4 and Appendix A.8). Below, we address each concern in the revised paper.

1. Core Contribution and Systematic Comparison with Prior Benchmarks (W1)

CRM-Agent-Bench addresses a core gap in enterprise agent evaluation: enforcing rigid, programmatic business obligations throughout adaptive, multi-turn customer interactions. In real CRM workflows, a polite reply or a valid final database row is insufficient if the agent skips a prerequisite validation, omits an audit trail, or attempts an unauthorized action that happens to be blocked by the backend. Each task therefore pairs an on-policy simulated user with a strict conjunctive contract (Yi=∏c∈Dic(τi,xT)Y_i = \prod_{c \in D_i} c(\tau_i, x_T) in Section 3.3 and Appendix D.2, Figure 14)—evaluating required actions, argument predicates, execution order, terminal target/protected state, and prohibited attempts via deterministic code, while restricting the LLM judge strictly to user-visible communication (where it cannot override any programmatic failure).

To make these structural distinctions explicit at a glance, we summarize the evaluation mechanisms across the closest benchmarks below (documented in detail in Appendix L, Table 23, alongside Table 1):

BenchmarkAdaptive User SimAction & Arg VerificationExecution Order & PrerequisitesProhibited Attempt Catching (e.g., 403)Target & Protected StateDecoupled Comm. Judge
SOPBench✗ (Canned repeats)✓ (Rule-based)✓ (Helper graph)Partial (Invoked actions)✓ (Oracle DB)✗ (Not in pass gate)
EnterpriseOps-Gym✗ (Static prompt)✗ (Path-agnostic)✗ (Endpoint only)✗ (State SQL only)✓ (SQL checks)✗ (Not graded)
STATE-Bench✓ (LLM user)Partial (LLM-judged)Partial (LLM-judged)Partial (LLM-judged)✓ (State + diff)✗ (Sees full trace)
ToolSandbox✓ (On-policy)Partial (Milestone DAG)✓ (Milestone DAG)✓ (Minefields)Partial (DB guardrails)Partial (String/sim match)
AppWorld✗ (Autonomous)✗ (Path-agnostic)✗ (Endpoint only)✗ (State diff only)✓ (Allowed diff)Partial (Answer fields)
CRM-Agent-Bench✓ (On-policy)✓ (Programmatic)✓ (Programmatic)✓ (Programmatic)✓ (Programmatic)✓ (Strictly isolated)

Key Operational Distinctions:

  • vs. SOPBench: SOPBench programmatically verifies action permissibility, prerequisite helpers, and final database state, but relies on canned information repeats rather than an adaptive on-policy customer, and does not grade communication quality as part of its pass criterion. CRM-Agent-Bench enforces explicit business obligations across adaptive multi-turn interactions while separately grading declared communication requirements (Appendix L, Table 23).
  • vs. EnterpriseOps-Gym & AppWorld: Both verify endpoint outcomes (via SQL scripts or process-agnostic unit tests) and accept alternative action paths. CRM-Agent-Bench additionally gates the declared action trajectory—including exact invocation counts, argument bindings, prerequisite ordering (164 tasks and 419 ordered slots in Table 11, e.g., validate_transfer before execute_transfer in Figure 2b and Appendix E.1), and rejected forbidden attempts—which are critical when intermediate business procedures are mandatory.
  • vs. ToolSandbox: While ToolSandbox pairs an on-policy user with milestone DAGs and minefields, it scores trajectories via continuous similarity. CRM-Agent-Bench treats business obligations as strict Boolean predicates—if an audit call is missed or a forbidden tool is attempted, the entire transaction fails (Section 3.3, Appendix L).
  • vs. STATE-Bench: STATE-Bench is closest in combining a simulated user with state/requirement conjunction, and its existing-record diff (like AppWorld's allowed-change tests) provides broader collateral protection than our sparse declared paths. However, STATE-Bench evaluates non-state procedural and policy requirements using an LLM judge over the full trajectory. In contrast, CRM-Agent-Bench assigns all action, argument, ordering, and prohibited-attempt requirements to deterministic programmatic verifiers, and restricts the semantic judge strictly to the user instruction, criteria, and visible assistant replies (withholding tool traces and state; Appendix D.2). This prevents fluent rationalizations from masking procedural violations and preserves independent, auditable per-component verdicts for diagnosis.
AuthorsOpenReview ↗

Continue response

What Our Conjunctive Contracts Uniquely Reveal in Practice (Trace Evidence in Appendices E, H, and J.2):

  • Backend Containment vs. Intrinsic Compliance (Appendix H, Line 2786; Appendix J.2, Case A; Appendix J.3, Probe C04): In task policy_refuse_other_users_appointment_01 (Field Service workflow, Figure 21), a user asks to unschedule another customer's appointment (SA-0412). In the saved GPT-5.5 execution (Appendix J.2, Case A), the agent calls FSA_Unschedule_Appointment twice, is blocked by the backend with a 403 ownership rejection, and then politely informs the user that it cannot fulfill the request. Across all 16 systems in our runtime tool-availability census (Appendix H, Table 16), 13 systems attempt this prohibited call on the live tool inventory while 12 still pass the semantic communication check. An outcome-only benchmark sees an unchanged database (protected_state passes); a response-only benchmark sees a safe refusal. Our programmatic forbidden-argument check uniquely catches the prohibited attempt in the trace log, separating backend containment from intrinsic agent compliance.
  • Silent Process Violations (Appendix J.2, Case B; Payments workflow, Figure 23): In task payments_checkout_multi_payment_cumulative_01, the agent processes a USD 40 card payment and a USD 60 PayPal payment, issues two receipts, and accurately reports 1,000 cumulative loyalty points. However, it omits the two required log_transaction calls (audit_log: expected min length 2, got 0). Because our contract enforces the strict conjunction of communication, state, and required actions (Table 11 and Table 19), it fails the agent for producing an incomplete audit trail despite a valid payment state and accurate user-facing reply.

As we mentioned to Reviewer asAq, as researchers working closely with real-world enterprise workflows, our goal is to push agent evaluation beyond simplified sandbox tasks toward the operational realities of production deployment. For tool-using agents to transition from research prototypes to trustworthy commercial systems, benchmarks must faithfully reflect how real business processes actually operate: workflows are multi-turn and stateful, intermediate procedural rules (such as identity verification, pre-execution validation, and audit logging) are just as mandatory as final database states, and unauthorized attempts carry real operational risk even when blocked by backend guardrails. Grounded in practitioner-reviewed CRM applications and production-inspired agent recipes within resettable, self-contained environments (Section 4.1, Appendix D.4, Table 13), we pair these conjunctive contracts with operational viability metrics (latency, token workload, and cost across 16 systems) to answer a concrete deployment question: which agent system satisfies strict business obligations reliably within an operating budget?


2. Evaluator Validation: Human Agreement, Deterministic Replay Probes, and Manual Execution Audit (Q1, W3)

We have completed a stratified human agreement study with three independent human annotators for the semantic communication judge (Appendix I, Tables 17–18), deterministic replay probes of the programmatic evaluator (Appendix J.3, Table 21), and a blind manual business-execution audit on 16 diagnostic trajectories (Appendix K, Table 22).

A. Stratified Human Agreement Study (Appendix I, Tables 17 & 18)

Metric / Rater BreakdownResultNotes / Reference in Paper
Pairwise Human Agreement (P1/P2, P1/P3, P2/P3)33/36, 29/36, 34/36Rotating pairs of 36 decisions each (Appendix I, Table 17)
Overall Human Inter-Rater Agreement (Response Verdict)88.9% (96 / 108)Pooled Cohen's κ=0.78\kappa = 0.78; 95% cluster bootstrap CI: [83.2%, 94.3%] (Table 17)
Human Criterion-Level Agreement88.9% (120 / 135)Pooled Cohen's κ=0.77\kappa = 0.77 across 135 criteria (Table 17)
Human Contradiction-Flag Agreement87.0% (94 / 108)Pooled Cohen's κ=0.40\kappa = 0.40 (low prevalence of contradiction flags; Table 17)
Unresolved / Split Human Ratings11.1% (12 / 108)3 on Judge-Pass (57 total), 9 on Judge-Fail (51 total) (Table 18)
Judge vs. Consensus Human Verdict95.8% (92 / 96)Measured on the 96 decisions where both human raters agreed (Table 18)
Judge-Pass / Both-Humans-Pass vs. Both-Humans-Fail54 Pass / 0 Fail0.0% false positive rate (0/57 overall; 0/54 excluding 3 split decisions; Table 18)
Judge-Fail / Both-Humans-Fail vs. Both-Humans-Pass38 Fail / 4 Pass7.8% (4/51 overall; 9.5%, 4/42 excluding 9 splits); logged for third-person human adjudication (Table 18)
AuthorsOpenReview ↗

Continue Response

Three human annotators provided independent ratings on 108 saved semantic-communication decisions covering 80 tasks and 135 criteria across all 9 domains and six evaluated systems (12 decisions per domain, stratified by domain and saved verdict, and failure-enriched with 57 judge passes and 51 judge fails). Each decision received two independent human ratings in rotating pairs (P1/P2, P1/P3, P2/P3) using the exact same inputs provided to the semantic judge (task instruction, response criteria, and visible assistant replies, withholding tool traces and database state; Appendix I, Lines 2817–2832):

B. Deterministic Replay Probes and Manual Business-Execution Audit (Appendix J.3 & Appendix K)

To test the full programmatic grader on valid alternatives, duplicate writes, and unintended state changes without LLM judge variance, we executed deterministic replay probes (Table 21) across fresh synthetic environments (rerun from clean copies with identical results and zero LLM or judge calls), further manually audited across 16 selected business executions in Appendix K (Table 22):

Probe / Audit ScopeTested Scenario & BaselineProgrammatic Grader & Manual Audit Outcome (Tables 21 & 22)
C01: reads (Valid alternatives)Extra read-only get_account_balance calls added before/between validate_transfer and execute_transferProgrammatic checks accept extra reads (subsequence matching, identical final balances) and reject reversed validate/execute order.
C04: prohibited attempt (Containment)Unauthorized FSA_Unschedule_Appointment returning 403 vs. permitted read returning 403Programmatic forbidden check catches and fails the 403 unschedule attempt (protected_state passes); permitted lookup returning 403 passes.
Duplicate writes on asserted paths (Figure 1; Case B, Appendix E.2)Repeated checkout in payments_checkout_full_chain_02 (claude-sonnet-4-6) and repeat runs (gemini-3.1-pro)Caught deterministically by expected_state (155 != 310 loyalty points in Case B; 123 != 246 in repeat runs) even when reply and action checks pass.
C02 & C03 (Sparse-contract boundary probes)C02: 3x USD 55 payments on a task without transaction-count state checks; C03: constructed extra USD 1 transfer on unasserted accountsProgrammatic checks accept both boundary probes because those specific state paths are unasserted (whereas mutating a declared protected loyalty path in C03 fails immediately).
Blind Manual Audit (Appendix K, Table 22)16 purposive diagnostic executions (locked prior to comparison)Manual acceptability labels match programmatic checks in 14 / 16 cases The only 2 divergences are the constructed boundary probes C02 and C03.
Reviewer_YwdROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CRM-Agent-Bench contains 555 tasks across nine synthetic business domains. Each task declares an executable contract over applicable action, argument, ordering, response, state, and forbidden-effect checks. An on-policy user simulator interacts with a routed agent over resettable tools. Sixteen systems achieve 64.32–83.60% strict success. For four inspected systems, response-only grading exceeds strict success by 10.5–11.0 points. The benchmark contributes portable enterprise workflows and separate reporting of contract satisfaction, policy checks, coverage, latency, and cost.

Critical strengths
  1. The benchmark makes a useful practical distinction between plausible communication and correct execution. The resettable environments, task-specific contracts, and separation from the reference agent support reuse. Counting forbidden attempts as violations, while distinguishing them from persistent state changes, is a sensible design for enterprise workflows.

  2. Reporting is careful about what was actually measured. All sixteen systems score all 555 tasks; Table 6 states which checks apply; and costs exclude simulator/judge usage and do not treat self-hosted models as free. The policy analysis separates negative-check compliance (91.2%) from full completion (66.8%).

Critical weaknesses
  1. Independent validation of the measurement is limited. The current semantic judge affects 402 tasks, but no human agreement study for that judge is reported. Gold-path replay establishes that a reference solution passes, not that alternative valid workflows pass or plausible incorrect workflows fail. Appendix B.4 also distinguishes early practitioner review from the mainly executable validation of the expanded set. A small independent audit of current contracts and judge decisions would strengthen both the strict scores and the claimed response-only gap. The acknowledged legacy judge error in Appendix C.2 motivates this check, but does not establish an error in the current judge.

  2. The comparisons use one trial per task with one simulator and one scaffold. The top eight systems are within 2.88 points, so their fine ordering and Pareto membership may change under repeated interaction. The paper appropriately calls the results descriptive, but this still limits model-selection conclusions. Repeat a representative subset for several seeds and report paired uncertainty. Also, graph-wide reachability of forbidden tools does not establish exposure on the turns where a violation was possible; runtime exposure would sharpen the policy analysis.

Questions for authors
  1. On a stratified sample from the final release, how often do independent reviewers disagree with the task contract or semantic judge?
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer YwdR: Human Judge Agreement, Deterministic Contract Probes, 3-Round & Cross-Simulator Reliability (19,980 Runs), and Runtime Exposure Audit

We thank Reviewer YwdR for the positive assessment and for the constructive suggestions on measurement validation, multi-run uncertainty, and runtime policy exposure. In the revised manuscript, we have added a stratified human agreement study (Appendix I), deterministic contract probes and a blind manual business-execution audit (Appendices J.3 and K), a decomposed 402-task contingency analysis (Figure 4 and Appendix A.8), three-round full-suite evaluations across all 16 systems (Appendix B), a 19,980-run user-simulator sensitivity study (Appendix C), and a turn-level runtime tool-availability census (Appendix H). Below, we address each point in detail.


1. Independent Validation of the Semantic Judge and Task Contracts (W1.1, W1.2, Q1)

A. Stratified Human Agreement Study on the Semantic Judge (Response to Q1; Appendix I)

We completed a stratified human agreement study on the current semantic-communication judge and added the full results to Appendix I (Tables 17–18). Three human annotators independently rated 108 saved judge decisions (spanning 80 tasks, 135 criteria, and all 9 domains across 6 systems, stratified by domain and saved verdict, and failure-enriched with 57 judge passes and 51 judge fails) in rotating pairs (two independent human ratings per decision). Annotators viewed the exact same inputs provided to the semantic judge (task instruction, response criteria, and visible assistant replies, withholding tool traces and database state):

Metric (Appendix I, Tables 17 & 18)ResultNotes / Denominator
Human Inter-Rater Agreement (Response Verdict)88.9% (96 / 108)Pooled Cohen's κ=0.78\kappa = 0.78; 95% cluster bootstrap CI: [83.2%, 94.3%] (Table 17)
Human Criterion-Level Agreement88.9% (120 / 135)Pooled Cohen's κ=0.77\kappa = 0.77 across 135 individual criteria (Table 17)
Unresolved / Split Human Ratings11.1% (12 / 108)Two human raters disagreed on the derived response verdict (Table 18)
Judge vs. Consensus Human Verdict95.8% (92 / 96)Measured on the 96 decisions where both human raters agreed (Table 18)
Judge-Pass / Both-Humans-Fail0.0% (0 / 57)0 / 54 excluding 3 split human decisions (Table 18)
Judge-Fail / Both-Humans-Pass7.8% (4 / 51)9.5% (4 / 42) excluding 9 split decisions; logged for third-person human adjudication (Appendix I)

Because the sample is stratified and failure-enriched by saved verdict, we report these figures as sample-level diagnostics rather than unweighted suite-wide prevalence rates.

B. Testing Full Scoring Rules on Alternatives, Duplicates, and Unintended Changes (Response to W1.1 & W1.2; Appendices J.3 & K)

We agree that gold-path replay primarily establishes executability and internal consistency. To test whether valid alternative workflows pass and plausible incorrect workflows fail, we executed a four-case deterministic replay suite across fresh synthetic environments (C01–C04 in Appendix J.3, Table 21, with zero LLM or judge calls) and conducted a blind manual business-execution audit across 16 selected trajectories (Appendix K, Table 22):

AuthorsOpenReview ↗

Continue Response

Probe / Audit ScopeScenario & Counterexample TestedProgrammatic Grader & Manual Audit Outcome (Appendix J.3, Table 21 & Appendix K, Table 22)
C01: reads (Valid alternatives vs. order violations)Authorized USD 100 transfer: inserting extra read-only get_account_balance lookups before/between validate_transfer and execute_transfer vs. reversing orderExtra read queries pass all programmatic checks (via subsequence matching) and reach the identical final state; executing transfer before validation reaches the same balances but is rejected by the strict order check.
C04: prohibited attempt (Isolating prohibited intent)Attempting to unschedule another customer's appointment (SA-0412) returning 403 vs. a permitted read lookup returning 403The unauthorized unschedule call fails the forbidden check even though the backend rejects it with 403 and leaves the database unchanged (protected_state passes). Conversely, a permitted read returning 403 does not trigger a violation.
Catching duplicate writes on asserted pathsRepeated checkout in payments_checkout_full_chain_02 (Figure 1; Case B in Appendix E.2) and repeat runs (gemini-3.1-pro on a USD 12.34 checkout)When an agent repeats a checkout after the user asks only for confirmation, it produces two transactions (310 loyalty points instead of 155 in Case B; 246 points instead of 123 in repeat runs). Even when the communication judge accepts the plausible reply, expected_state catches and fails the duplicate write.
C02 & C03 (Sparse-contract boundary probes)C02: replaying an archived checkout into three synthetic USD 55 payment records where payment multiplicity is unasserted; C03: constructed USD 1 transfer between unasserted accountsBoth probes pass the sparse programmatic checks because those specific state paths are not declared by the task contract (whereas mutating a declared protected loyalty path in C03 fails immediately). Similarly, in human-reviewed item H021 (improper weekend rebooking where protected state only checked that status not end as Unscheduled), the semantic response check caught the policy violation (confirmed by both human raters).
Blind Manual Audit (Appendix K, Table 22)16 diagnostic executions (13 replays from C01–C04 + 3 interactive traces across Payments and Field Service; labels locked before revealing programmatic outcomes)Manual business-acceptability labels match programmatic contract checks in 14 / 16 cases (8 acceptable, 6 unacceptable; 0 programmatic rejections deemed acceptable). The only 2 divergences are the constructed sparse-boundary probes C02 and C03.

C. Manual Practitioner Review Across the Expanded Task Set (Response to W1.2; Appendix D.4)

Regarding task validation in Appendix D.4 (formerly Appendix B.4), we appreciate your close reading. The earlier manuscript text understated our manual validation effort: in addition to source-to-contract and fresh-environment gold-path checks, we conducted manual human reviews across the expanded task set to verify logical consistency between user goals, domain policies, and contract predicates. We have updated Appendix D.4 to document this review process alongside the scope boundaries of sparse contracts (state assertions apply to 200/555 tasks, including 45 with protected-state checks; Section 5.1 and Appendix D.2, Table 11).

AuthorsOpenReview ↗

Continue Response


2. Clarifying the Response-Only Gap and Legacy Judge Behavior (W1.3)

We agree that the response-only gap requires explicit task denominators, and that the single-trace judge false negative noted in Appendix E.2 (formerly Appendix C.2, Case B) motivates clear separation between legacy (v4/v5 in our previous pipeline) full-suite relaxation and matched evaluation on the 402 response-declared tasks.

In the revised manuscript, we updated Figure 4 (Section 5.2) and Figure 12 (Appendix A.8) to evaluate strictly the matched 402 tasks that declare semantic response criteria, and added the complete 16-system decomposed contingency table in Appendix A.8 (Table 5) alongside component-level failure attribution in Appendix J.1 (Tables 19–20):

System (Excerpt from Table 5, Appendix A.8)R+/S+R^+ / S^+ (Pass Both)R+/S−R^+ / S^- (Matched 402 Gap)R−/S+R^- / S^+R−/S−R^- / S^- (Fail Both)R?/S−R^? / S^- (Missing Resp.)Other 153 Tasks (S−S^-)Relaxed Full-555 Gap GG
Gemini 3.8 Flash329 (81.8%)42 (10.45%)03101860 (10.81%)
Claude Opus 5325 (80.8%)38 (9.45%)03811958 (10.45%)
GPT-5.6 Sol319 (79.4%)38 (9.45%)04502159 (10.63%)
Muse Glimmer 30B318 (79.1%)27 (6.72%)05703461 (10.99%)
All 16 Systems Total (8,880 runs, Table 5)4,942650 (27–59 / sys)082317501 (17–59 / sys)1,168 (56–118 / sys)

Key takeaways from this decomposition (Section 5.2 and Appendices A.8 & J.1):

  • Matched 402-Task Scope vs. Relaxed 555-Task Scope: The 10.45–10.99 point gap originally quoted across all 555 tasks (G=58 to 61G = 58\text{ to }61 tasks) combined observed deceptive passes on the 402 response-declared tasks (R+/S−=27 to 42R^+ / S^- = 27\text{ to }42 tasks, or 6.72–10.45 percentage points on the matched 402 denominator) with strict failures on the 153 tasks that declare no response criteria (Other153(S−)=18 to 34\text{Other}_{153}(S^-) = 18\text{ to }34 tasks, treated as neutral passes under a response-only rule).
  • Persistence Under Current v6 Grading (Appendix J.1, Tables 19–20): Re-evaluating the uniform 6-agent R1 cohort under semantic-communication-current (with zero missing verdicts; Table 20) yields an identical pattern: 30 to 47 of the 402 response-declared executions per agent (252 / 2,195 accepted replies overall, or 11.5%) pass current communication but fail strict execution, with required-action omissions accounting for 81.7% (206 / 252) of those failures (Table 19). Crucially, R−/S+=0R^- / S^+ = 0 across all runs, confirming that the gap is driven by plausible replies accompanying incomplete or invalid execution rather than judge leniency artifacts.

3. Multi-Round Repeatability, Paired Uncertainty, and Cross-Simulator Sensitivity (W2.1)

A. Three-Round Full-Suite Evaluation Across All 16 Systems (Appendix B, Table 7)

Rather than repeating only a subset of tasks, we executed all 16 leaderboard systems for three complete rounds across the 555-task suite with distinct framework seeds, reporting equal-round means, sample standard deviations, rank correlations, paired uncertainty, and the all-three-success reliability metric pass^3\widehat{\text{pass}}^3 in Appendix B (Table 7):

Reviewer_asAqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CRM-Agent-Bench, a benchmark for evaluating tool-using enterprise agents beyond response correctness. It contains 555 stateful, multi-turn tasks across nine business domains and evaluates agents using task-specific executable contracts covering tool actions, arguments, execution order, responses, state changes, and policy compliance. Experiments on 16 systems demonstrate that response-only evaluation can substantially overestimate task success and reveal differences in reliability and operational cost.

Critical strengths

The evaluation covers 16 systems across diverse enterprise workflows. The analysis of response-only versus strict success provides useful insights into agent failures that conventional metrics may overlook.

Critical weaknesses
  1. There is insufficient differentiation from existing agent benchmarks. The proposed benchmark substantially overlaps with existing work, particularly SOPBench, STATE-Bench, and EnterpriseOps-Gym. SOPBench already evaluates procedural compliance, tool-use correctness, and database outcomes through executable verifiers. STATE-Bench supports multi-turn enterprise workflows with simulated users, stateful environments, and reliability and efficiency evaluation. Although CRM-Agent-Bench integrates these aspects through task-specific executable contracts, it remains unclear what fundamentally new evaluation capabilities the benchmark introduces. Furthermore, the paper does not adequately discuss or compare against several highly relevant benchmarks, particularly SOPBench and STATE-Bench. Without a more systematic comparison or evidence demonstrating the incremental value of its evaluation design, the contribution appears largely incremental.
  2. Each model is evaluated only once per task using a fixed seed, despite the benchmark relying on an LLM-based user simulator and semantic judge. The paper does not provide sufficient evidence of human agreement or sensitivity to different simulators and evaluation runs. Given the small performance differences among leading models, these limitations raise concerns about the stability and reliability of the reported rankings.
Questions for authors

How does CRM-Agent-Bench differ fundamentally from SOPBench and STATE-Bench, which already evaluate procedural compliance, tool execution, and stateful enterprise workflows? Can the authors provide a more systematic comparison and demonstrate which agent failure modes are uniquely captured by the proposed executable contracts?

7 comments
AuthorsOpenReview ↗

Response to Reviewer asAq: Mechanism Comparison, Sole-Check Failure Attribution, and Multi-Run / Simulator / Human Reliability Studies

We thank Reviewer asAq for the thorough, constructive feedback. We agree that a systematic mechanism comparison against closely related benchmarks and rigorous empirical validation across multiple evaluation runs, user simulators, and human annotators are essential to establish the benchmark's contributions. We have conducted extensive new experiments across all of these dimensions and integrated them—alongside a new mechanism comparison table—into the revised manuscript. Below, we address your questions and concerns in detail.


1. Core Contribution and Systematic Comparison with Prior Benchmarks (W1, Q1)

CRM-Agent-Bench addresses a core gap in enterprise agent evaluation: enforcing rigid, programmatic business obligations throughout adaptive, multi-turn customer interactions. In real CRM workflows, a polite reply or a valid final database row is insufficient if the agent skips a prerequisite validation, omits an audit trail, or attempts an unauthorized action that happens to be blocked by the backend. Each task therefore pairs an on-policy simulated user with a strict conjunctive contract—evaluating required actions, argument predicates, execution order, terminal target/protected state, and prohibited attempts via deterministic code, while restricting the LLM judge strictly to user-visible communication (where it cannot override any programmatic failure).

To make these structural distinctions explicit at a glance, we summarize the evaluation mechanisms across the closest benchmarks below (documented in detail in Appendix L, Table 23, and being incorporated into the main text):

BenchmarkAdaptive User SimAction & Arg VerificationExecution Order & PrerequisitesProhibited Attempt Catching (e.g., 403)Target & Protected StateDecoupled Comm. Judge
SOPBench✗ (Canned repeats)✓ (Rule-based)✓ (Helper graph)Partial (Invoked actions)✓ (Oracle DB)✗ (Not in pass gate)
EnterpriseOps-Gym✗ (Static prompt)✗ (Path-agnostic)✗ (Endpoint only)✗ (State SQL only)✓ (SQL checks)✗ (Not graded)
STATE-Bench✓ (LLM user)Partial (LLM-judged)Partial (LLM-judged)Partial (LLM-judged)✓ (State + diff)✗ (Sees full trace)
ToolSandbox✓ (On-policy)Partial (Milestone DAG)✓ (Milestone DAG)✓ (Minefields)Partial (DB guardrails)Partial (String/sim match)
AppWorld✗ (Autonomous)✗ (Path-agnostic)✗ (Endpoint only)✗ (State diff only)✓ (Allowed diff)Partial (Answer fields)
CRM-Agent-Bench✓ (On-policy)✓ (Programmatic)✓ (Programmatic)✓ (Programmatic)✓ (Programmatic)✓ (Strictly isolated)

Key Operational Distinctions:

  • vs. SOPBench: SOPBench programmatically verifies action permissibility, prerequisite helpers, and final database state, but relies on canned information repeats rather than an adaptive on-policy customer, and does not grade communication quality as part of its pass criterion. CRM-Agent-Bench enforces explicit business obligations across adaptive multi-turn interactions while separately grading declared communication requirements.
  • vs. EnterpriseOps-Gym & AppWorld: Both verify endpoint outcomes (via SQL scripts or process-agnostic unit tests) and accept alternative action paths. CRM-Agent-Bench additionally gates the declared action trajectory—including exact invocation counts, argument bindings, prerequisite ordering, and rejected forbidden attempts—which are critical when intermediate business procedures (e.g., validation before execution) are mandatory.
  • vs. STATE-Bench: STATE-Bench is closest in combining a simulated user with state/requirement conjunction, and its existing-record diff (like AppWorld's allowed-change tests) provides broader collateral protection than our sparse declared paths. However, STATE-Bench evaluates non-state procedural and policy requirements using an LLM judge over the full trajectory. In contrast, CRM-Agent-Bench assigns all action, argument, ordering, and prohibited-attempt requirements to deterministic programmatic verifiers, and restricts the semantic judge strictly to the user instruction, criteria, and visible assistant replies (withholding tool traces and state). This prevents fluent rationalizations from masking procedural violations and preserves independent, auditable per-component verdicts for diagnosis (Appendix D).
AuthorsOpenReview ↗

Continue response

As researchers working closely with real-world enterprise workflows, our goal is to push agent evaluation beyond simplified sandbox tasks toward the operational realities of production deployment. For tool-using agents to transition from research prototypes to trustworthy commercial systems, benchmarks must faithfully reflect how real business processes actually operate: workflows are multi-turn and stateful, intermediate procedural rules (such as identity verification, pre-execution validation, and audit logging) are just as mandatory as final database states, and unauthorized attempts carry real operational risk even when blocked by backend guardrails. Grounded in practitioner-reviewed CRM applications and production-inspired agent recipes within resettable, self-contained environments (Section 4.1), CRM-Agent-Bench pairs these strict workflow contracts with operational viability metrics (latency, token workload, and cost across 16 systems) to help the community measure and build agents that are not merely conversational, but genuinely deployable in real business processes.


2. Unique Failure Modes Captured by Executable Contracts (W1, Q2)

Our executable contracts decouple evaluation into distinct programmatic and semantic predicates—separating what the agent said (semantic response), what it did (required actions, arguments, ordering, and forbidden attempts), and what it changed (target and protected state). Combining our suite-wide failure composition (Appendix A.2, Figures 7–8; Appendix A.8, Table 5) with a new sole-failing check attribution and controlled replay analysis (Appendix J, pp. 55–57, Tables 19–21) demonstrates that each check family isolates distinct, non-redundant failure modes:

A. Suite-Wide Trajectory Bottlenecks (Appendix A.2 & A.8: 16-System Main Cohort)

Across all 16 systems (8,880 executions), action mismatch is consistently the single largest cause of strict failure (54 to 157 tasks per system), whereas primary semantic-response failures are comparatively rare (14 to 34 tasks). On the 402 response-declared tasks, grading responses alone would falsely pass 27 to 59 executions per system (650 executions total across the 16 systems) that fail at least one programmatic execution requirement (Table 5).

B. Non-Redundant Check Attribution (Appendix J.1, Tables 19–20: 6-Agent Uniform Luna Cohort, 3,330 Executions)

In our uniform 6-agent diagnostic cohort (Round 1 under GPT-6 Luna and the current communication judge), 11.5% (252 / 2,195) of accepted replies still fail the executable contract (30–47 tasks per agent). Out of 592 total strict failures, 257 (43.4%) are caused by a single isolated check (147 sole-execution failures: 103 with an accepted reply and 44 on tasks without response criteria, plus 110 sole-response failures, alongside 335 multi-component failures), where removing that single predicate flips the recorded task verdict to a pass:

Contract Check FamilyApplicable Tasks (/555)Sole-Failing Count (of 257)Per-Agent Sole RangeDiagnostic Role & Non-Redundancy
Required Actions529866–18Catches omitted or under-repeated operational steps when replies sound complete
Expected Arguments493242–6Catches wrong entity/parameter bindings despite calling the right tool
Forbidden Calls / Args136202–5Catches unauthorized attempts even when blocked by the backend (e.g., HTTP 403)
Action Ordering164161–6Catches prerequisite violations (e.g., executing a mutation before validating)
Expected Target State18010–1Catches state-level discrepancies missed by trace/reply checks (49 others co-occur)
Protected State4500Acts as a state-level safety net; violations co-occur with forbidden/action checks
Semantic Response40211011–25Catches missing explanations, policy errors, or contradictions when tool calls succeed

For every single agent in this cohort, required actions, arguments, ordering, forbidden operations, and communication checks each independently decide outcomes on their own (Table 19).

AuthorsOpenReview ↗

Continue response

C. Concrete Trajectory Examples & Controlled Boundary Probes (Appendix J.2 & J.3, Table 21)

Three archived GPT-5.5 trajectories (Appendix J.2) and four shared programmatic control probes (Appendix J.3, Table 21) illustrate specific failure modes that outcome-only, response-only, or trajectory-LLM evaluators miss:

  1. Backend Containment vs. Agent Compliance (policy_refuse_other_users_appointment_01): Asked to reschedule another customer's appointment (SA-0412), GPT-5.5 called FSA_Unschedule_Appointment twice; both attempts were blocked by backend 403 ownership rejections before the agent refused the user. Because no state mutated and the final refusal was polite, both the protected-state check and the communication judge passed—yet our programmatic forbidden-argument check caught the two unauthorized attempts (see also the 16-system runtime tool-availability census in Appendix H, Table 16, where 13/16 systems attempt the prohibited call and 12/13 pass the communication check).
  2. Endpoint Outcome vs. Workflow Integrity (payments_checkout_multi_payment_cumulative_01): The agent executed two valid payments (USD 40 card and USD 60 PayPal), issued two receipts, and awarded 1,000 cumulative loyalty points—passing all argument and communication checks—but omitted the two required log_transaction calls. Our required-action, ordering, and state (audit_log) checks rejected the incomplete workflow despite valid financial balances and replies.
  3. Exact Predicate Enforcement (compatibility_check_01): The agent invoked the required CSA_Product_Compatibility tool and produced a accepted reply, but queried RelationshipType=compatible instead of the required incompatible predicate, which was isolated solely by the expected-argument check.
  4. Controlled Replays & Coverage Boundaries (Appendix J.3, Table 21): Deterministic replay probes (C01–C04) verify that benign read-only interleavings pass while reversing required validation-before-execution order fails despite identical final balances (C01); permitted lookups returning 403 pass while prohibited unschedule attempts returning 403 fail (C04); and sparse state checks reject mutations to declared protected paths while transparently documenting the boundary of unasserted paths (C02–C03, further audited in Appendix K, Table 22).

3. Stability Across Runs, Simulator Sensitivity, and Human Agreement (W2)

We thank the reviewer for emphasizing these three pillars of evaluation reliability. We have completed dedicated empirical studies across all three fronts and integrated them into the revised manuscript:

A. Multi-Round Evaluation Stability (Appendix B, Table 7)

Rather than relying on a single trial, we evaluated multiple complete rounds on the 555-task canonical set using distinct framework seeds (Appendix B, Table 7). Below is the three-round breakdown for the top two overall leaders alongside the three systems that tied at 451/555 (81.26%) in Round 1:

Agent ModelRound 1 (Pass/555)Round 2 (Pass/555)Round 3 (Pass/555)3-Round Mean ± SDAll-3-Pass (pass^3\widehat{\text{pass}}^3)
Gemini 3.8 Flash464 (83.60%)480 (86.49%)480 (86.49%)85.53% ± 1.6678.20%
Claude Opus 5459 (82.70%)462 (83.24%)465* (83.94%)83.29% ± 0.6273.47%
GPT-5.5 (R1 tie)451 (81.26%)462 (83.24%)465 (83.78%)82.76% ± 1.3374.77%
GPT-5.6 Sol (R1 tie)451 (81.26%)457 (82.34%)451 (81.41%)81.67% ± 0.5973.65%
Claude Sonnet 5 (R1 tie)451 (81.26%)445 (80.18%)441 (79.46%)80.30% ± 0.9170.45%
  • High Macro-Level Rank Stability: Across all 16 systems, pairwise Spearman rank correlations between rounds are 0.922 (R1–R2), 0.869 (R1–R3), and 0.934 (R2–R3) (and 0.869, 0.827, 0.917 on the preserved 13-system historical subset in Appendix B.2). Gemini 3.8 Flash ranks 1st in every round, and Claude Opus 5 remains 2nd overall.
  • Transparent Framing of Fine Ordering: As the reviewer noted, fine-grained positions within the tightly compressed top group (ranks 1–8 spanning 2.88 pp in R1) fluctuate across runs, and among the three systems tied at 451/555 in Round 1, no pairwise difference survives Holm correction across repeats (moreover, failure-set Jaccard overlap among the tied trio is only 31.6%–45.5%, Appendix B.4). In the revised manuscript (Section 5 & Appendix B), we explicitly frame fine ordering within this top cluster as descriptive and report multi-run means, standard deviations, paired uncertainty, and pass^3\widehat{\text{pass}}^3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.