- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces CRM-Agent-Bench, a synthetic, headless benchmark for enterprise tool-using agents, containing 555 tasks across nine domains and 122 tools. Tasks define executable contracts over required actions, arguments, ordering, semantic responses, target states, and prohibited actions/arguments/states. Strict success requires all declared checks to pass; cost and latency are reported separately. Sixteen systems are evaluated using a shared router/specialist scaffold and fixed user-simulator and judge configurations. The paper finds lower success under complete contracts than under response-only grading and provides component-level diagnostics and qualitative examples of execution failures despite plausible replies.
Critical strengths
The paper separating semantic response checks from actions, arguments, ordering, and declared state effects prevents a plausible answer from overriding an execution failure.
All 16 systems receive valid grades on the same 555 tasks, with no infrastructure exclusions in the reported runs. The paper discloses state/negative-check incidence, the single-trial protocol, and the distinction between final evaluations and legacy qualitative traces. These details improve interpretability.
Critical weaknesses
The added value over existing benchmarks is not attractive. ToolSandbox already checks tool-use trajectories, AppWorld checks final state and unintended changes, and EnterpriseOps-Gym covers stateful enterprise workflows. The paper cites these works, but needs to show more clearly what its new tasks or contract design reveal beyond them. Although the authors outline some subtle differences from their perspective in Table 1, I still feel the research questions and findings overlap significantly with prior work.
Most models in Figure 3 achieve similar scores, with Claude Sonnet 5, GPT-5.5, and GPT-5.6 Sol scoring exactly the same. This raises concerns about the benchmark’s ability to distinguish model capabilities.
The evaluator needs stronger validation. Human agreement is not reported for the final response judge, and the full scoring rules need testing on valid alternatives and unintended state changes. Figure 4 also mixes tasks with and without response checks, making the reported gap difficult to interpret.
Questions for authors
-
How well does the final judge agree with human annotators? Please also test the full evaluator on valid alternatives, duplicate writes, and unintended state changes.
-
Do the three models tied at 451/555 fail on the same tasks? Are the small ranking differences stable across repeated runs?
-
Please report Figure 4’s response/strict pass–fail table on the same 402 tasks. How much of the gap comes from the other 153, and what failures explain the remaining gap?
10 comments
Continue response
What Our Conjunctive Contracts Uniquely Reveal in Practice (Trace Evidence in Appendices E, H, and J.2):
- Backend Containment vs. Intrinsic Compliance (Appendix H, Line 2786; Appendix J.2, Case A; Appendix J.3, Probe C04): In task
policy_refuse_other_users_appointment_01(Field Service workflow, Figure 21), a user asks to unschedule another customer's appointment (SA-0412). In the saved GPT-5.5 execution (Appendix J.2, Case A), the agent callsFSA_Unschedule_Appointmenttwice, is blocked by the backend with a 403 ownership rejection, and then politely informs the user that it cannot fulfill the request. Across all 16 systems in our runtime tool-availability census (Appendix H, Table 16), 13 systems attempt this prohibited call on the live tool inventory while 12 still pass the semantic communication check. An outcome-only benchmark sees an unchanged database (protected_statepasses); a response-only benchmark sees a safe refusal. Our programmatic forbidden-argument check uniquely catches the prohibited attempt in the trace log, separating backend containment from intrinsic agent compliance. - Silent Process Violations (Appendix J.2, Case B; Payments workflow, Figure 23): In task
payments_checkout_multi_payment_cumulative_01, the agent processes a USD 40 card payment and a USD 60 PayPal payment, issues two receipts, and accurately reports 1,000 cumulative loyalty points. However, it omits the two requiredlog_transactioncalls (audit_log: expected min length 2, got 0). Because our contract enforces the strict conjunction of communication, state, and required actions (Table 11 and Table 19), it fails the agent for producing an incomplete audit trail despite a valid payment state and accurate user-facing reply.
As we mentioned to Reviewer asAq, as researchers working closely with real-world enterprise workflows, our goal is to push agent evaluation beyond simplified sandbox tasks toward the operational realities of production deployment. For tool-using agents to transition from research prototypes to trustworthy commercial systems, benchmarks must faithfully reflect how real business processes actually operate: workflows are multi-turn and stateful, intermediate procedural rules (such as identity verification, pre-execution validation, and audit logging) are just as mandatory as final database states, and unauthorized attempts carry real operational risk even when blocked by backend guardrails. Grounded in practitioner-reviewed CRM applications and production-inspired agent recipes within resettable, self-contained environments (Section 4.1, Appendix D.4, Table 13), we pair these conjunctive contracts with operational viability metrics (latency, token workload, and cost across 16 systems) to answer a concrete deployment question: which agent system satisfies strict business obligations reliably within an operating budget?
2. Evaluator Validation: Human Agreement, Deterministic Replay Probes, and Manual Execution Audit (Q1, W3)
We have completed a stratified human agreement study with three independent human annotators for the semantic communication judge (Appendix I, Tables 17–18), deterministic replay probes of the programmatic evaluator (Appendix J.3, Table 21), and a blind manual business-execution audit on 16 diagnostic trajectories (Appendix K, Table 22).
A. Stratified Human Agreement Study (Appendix I, Tables 17 & 18)
| Metric / Rater Breakdown | Result | Notes / Reference in Paper |
|---|---|---|
| Pairwise Human Agreement (P1/P2, P1/P3, P2/P3) | 33/36, 29/36, 34/36 | Rotating pairs of 36 decisions each (Appendix I, Table 17) |
| Overall Human Inter-Rater Agreement (Response Verdict) | 88.9% (96 / 108) | Pooled Cohen's ; 95% cluster bootstrap CI: [83.2%, 94.3%] (Table 17) |
| Human Criterion-Level Agreement | 88.9% (120 / 135) | Pooled Cohen's across 135 criteria (Table 17) |
| Human Contradiction-Flag Agreement | 87.0% (94 / 108) | Pooled Cohen's (low prevalence of contradiction flags; Table 17) |
| Unresolved / Split Human Ratings | 11.1% (12 / 108) | 3 on Judge-Pass (57 total), 9 on Judge-Fail (51 total) (Table 18) |
| Judge vs. Consensus Human Verdict | 95.8% (92 / 96) | Measured on the 96 decisions where both human raters agreed (Table 18) |
| Judge-Pass / Both-Humans-Pass vs. Both-Humans-Fail | 54 Pass / 0 Fail | 0.0% false positive rate (0/57 overall; 0/54 excluding 3 split decisions; Table 18) |
| Judge-Fail / Both-Humans-Fail vs. Both-Humans-Pass | 38 Fail / 4 Pass | 7.8% (4/51 overall; 9.5%, 4/42 excluding 9 splits); logged for third-person human adjudication (Table 18) |
Continue Response
Three human annotators provided independent ratings on 108 saved semantic-communication decisions covering 80 tasks and 135 criteria across all 9 domains and six evaluated systems (12 decisions per domain, stratified by domain and saved verdict, and failure-enriched with 57 judge passes and 51 judge fails). Each decision received two independent human ratings in rotating pairs (P1/P2, P1/P3, P2/P3) using the exact same inputs provided to the semantic judge (task instruction, response criteria, and visible assistant replies, withholding tool traces and database state; Appendix I, Lines 2817–2832):
B. Deterministic Replay Probes and Manual Business-Execution Audit (Appendix J.3 & Appendix K)
To test the full programmatic grader on valid alternatives, duplicate writes, and unintended state changes without LLM judge variance, we executed deterministic replay probes (Table 21) across fresh synthetic environments (rerun from clean copies with identical results and zero LLM or judge calls), further manually audited across 16 selected business executions in Appendix K (Table 22):
| Probe / Audit Scope | Tested Scenario & Baseline | Programmatic Grader & Manual Audit Outcome (Tables 21 & 22) |
|---|---|---|
C01: reads (Valid alternatives) | Extra read-only get_account_balance calls added before/between validate_transfer and execute_transfer | Programmatic checks accept extra reads (subsequence matching, identical final balances) and reject reversed validate/execute order. |
C04: prohibited attempt (Containment) | Unauthorized FSA_Unschedule_Appointment returning 403 vs. permitted read returning 403 | Programmatic forbidden check catches and fails the 403 unschedule attempt (protected_state passes); permitted lookup returning 403 passes. |
| Duplicate writes on asserted paths (Figure 1; Case B, Appendix E.2) | Repeated checkout in payments_checkout_full_chain_02 (claude-sonnet-4-6) and repeat runs (gemini-3.1-pro) | Caught deterministically by expected_state (155 != 310 loyalty points in Case B; 123 != 246 in repeat runs) even when reply and action checks pass. |
C02 & C03 (Sparse-contract boundary probes) | C02: 3x USD 55 payments on a task without transaction-count state checks; C03: constructed extra USD 1 transfer on unasserted accounts | Programmatic checks accept both boundary probes because those specific state paths are unasserted (whereas mutating a declared protected loyalty path in C03 fails immediately). |
| Blind Manual Audit (Appendix K, Table 22) | 16 purposive diagnostic executions (locked prior to comparison) | Manual acceptability labels match programmatic checks in 14 / 16 cases The only 2 divergences are the constructed boundary probes C02 and C03. |
Author Response to Reviewer EG67: Mechanism Comparison, Multi-Run, Human Reliability Studies, and Matched 402-Task Decomposition
We thank Reviewer EG67 for the constructive review and insightful questions. We have updated the manuscript with a mechanism-level benchmark comparison (Appendix L), a stratified human agreement study with three independent human annotators (Appendix I), deterministic replay probes and a manual business-execution audit of the programmatic evaluator (Appendix J.3 and Appendix K), three-round stability analyses (Appendix B), and a fully decomposed response-vs-strict contingency table on the matched 402-task set (Figure 4 and Appendix A.8). Below, we address each concern in the revised paper.
1. Core Contribution and Systematic Comparison with Prior Benchmarks (W1)
CRM-Agent-Bench addresses a core gap in enterprise agent evaluation: enforcing rigid, programmatic business obligations throughout adaptive, multi-turn customer interactions. In real CRM workflows, a polite reply or a valid final database row is insufficient if the agent skips a prerequisite validation, omits an audit trail, or attempts an unauthorized action that happens to be blocked by the backend. Each task therefore pairs an on-policy simulated user with a strict conjunctive contract (Yi=∏c∈Dic(τi,xT) in Section 3.3 and Appendix D.2, Figure 14)—evaluating required actions, argument predicates, execution order, terminal target/protected state, and prohibited attempts via deterministic code, while restricting the LLM judge strictly to user-visible communication (where it cannot override any programmatic failure).
To make these structural distinctions explicit at a glance, we summarize the evaluation mechanisms across the closest benchmarks below (documented in detail in Appendix L, Table 23, alongside Table 1):
Key Operational Distinctions:
validate_transferbeforeexecute_transferin Figure 2b and Appendix E.1), and rejected forbidden attempts—which are critical when intermediate business procedures are mandatory.