- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a great APEX-Voice, a benchmark of 120 synthetic professional workflows (10 archetypes, e.g., form-fill, negotiation, coordination, interviewing) executed through full-duplex speech. The evaluation defines a conjunctive Workflow Success metric with four gates (terminal state, process validity, action completion, artifact validity) and complementary artifact field accuracy, and reports that five frontier voice systems achieve ≤25% PASS@1 with low RELIABLE@3, revealing state-tracking and artifact-synthesis bottlenecks—especially under mid-speech corrections—despite strong floor-control behavior.
Critical strengths
- A well-motivated, executable evaluation design that measures outcomes rather than conversation quality.
- The four-gate Workflow Success (TS, PV, AC, AV) paired with Artifact Field Accuracy separates local correctness from global deliverable validity and provides actionable diagnostics.
- Repeated-run reliability metrics (PASS@1, PASS@3, RELIABLE@3) foreground consistency—crucial for real deployment and often ignored in benchmarks.
Critical weaknesses
- Figure 1 caption vs. Table 2: the caption says no system exceeds “10% Reliable@3” (L024–025), but Table 2 reports 10.8% for GPT-realtime-2.1.
- Text control vs. Table 2: the “voice reference” for GPT-realtime-2.1 in Table 5 is 23.3, whereas Table 2 reports 23.6.
- Table 9 reports a leakage rate of 4.1% with a 95% CI of [4.09%, 4.16%]. That interval is implausibly narrow for 72 utterances under a task-level bootstrap over 24 tasks.
- Figure 5(d) lists Healthcare (4) and no Insurance, while Table 11 has Healthcare n=3 and Insurance n=1.
- The text-only control (Table 5) and the cascaded baseline (Table 2) are never described;
Questions for authors
- How were the text-only controls and the cascaded baseline run (inputs, prompt, tools, number of runs)?
- How often is the LLM judge invoked, and how well does it agree with human graders?
- Is the kb_search requirement communicated to agents? How would Pass@1 change if AC only checked outcomes?
- For the human studies: how many participants were there, how were they recruited and paid, and was there consent/IRB review?
- Can you provide confidence intervals or bootstrap estimates for PASS@k and gate metrics? Are RELIABLE@3 differences across systems statistically significant?