acceptodds
Under review as a conference paper at ICLR 2027

APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction

Abstract

Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional workflows. We introduce APEX-Voice, a benchmark of 120 interactive professional workflows spanning ten work archetypes such as form completion, corporate negotiation, coordination, consulting, and interviewing. Each workflow executes in a stateful Voice Workbench environment with task-specific knowledge, typed tools, gold-annotated final work artifact, authorization constraints, and a user simulation policy backed by validated, pre-compiled speech realizations. We evaluate both artifact field accuracy and end-to-end workflow success, which requires the correct terminal state, valid process, completed actions, and a valid final artifact. Across five frontier real-time voice agents-GPT-Live-1, Gemini-3.8-Live, Grok-Voice-Think-2.0, Step-Audio3, and GPT-realtime-2.1, none exceeds 25% Pass@1, and the best Reliable@3 is only 10.8%. Moreover, stateful coordination is the dominant failure point across systems, while success decreases further on workflows requiring greater knowledge retrieval and mid-speech corrections. Overall, APEX-Voice is the first benchmark for evaluating whether voice agents can translate conversational competence into dependable professional work.

Primary area:
datasets and benchmarks
Keywords:
full-duplex, voice agents, agent evaluation, professional workflows, tool use, speech-language models, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Uf96OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a great APEX-Voice, a benchmark of 120 synthetic professional workflows (10 archetypes, e.g., form-fill, negotiation, coordination, interviewing) executed through full-duplex speech. The evaluation defines a conjunctive Workflow Success metric with four gates (terminal state, process validity, action completion, artifact validity) and complementary artifact field accuracy, and reports that five frontier voice systems achieve ≤25% PASS@1 with low RELIABLE@3, revealing state-tracking and artifact-synthesis bottlenecks—especially under mid-speech corrections—despite strong floor-control behavior.

Critical strengths
  1. A well-motivated, executable evaluation design that measures outcomes rather than conversation quality.
  2. The four-gate Workflow Success (TS, PV, AC, AV) paired with Artifact Field Accuracy separates local correctness from global deliverable validity and provides actionable diagnostics.
  3. Repeated-run reliability metrics (PASS@1, PASS@3, RELIABLE@3) foreground consistency—crucial for real deployment and often ignored in benchmarks.
Critical weaknesses
  1. Figure 1 caption vs. Table 2: the caption says no system exceeds “10% Reliable@3” (L024–025), but Table 2 reports 10.8% for GPT-realtime-2.1.
  2. Text control vs. Table 2: the “voice reference” for GPT-realtime-2.1 in Table 5 is 23.3, whereas Table 2 reports 23.6.
  3. Table 9 reports a leakage rate of 4.1% with a 95% CI of [4.09%, 4.16%]. That interval is implausibly narrow for 72 utterances under a task-level bootstrap over 24 tasks.
  4. Figure 5(d) lists Healthcare (4) and no Insurance, while Table 11 has Healthcare n=3 and Insurance n=1.
  5. The text-only control (Table 5) and the cascaded baseline (Table 2) are never described;
Questions for authors
  1. How were the text-only controls and the cascaded baseline run (inputs, prompt, tools, number of runs)?
  2. How often is the LLM judge invoked, and how well does it agree with human graders?
  3. Is the kb_search requirement communicated to agents? How would Pass@1 change if AC only checked outcomes?
  4. For the human studies: how many participants were there, how were they recruited and paid, and was there consent/IRB review?
  5. Can you provide confidence intervals or bootstrap estimates for PASS@k and gate metrics? Are RELIABLE@3 differences across systems statistically significant?
Reviewer_Q1JfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

this paper presents a benchmark for full-duplex voice based agents, combining metrics for voice interaction (barge-in, speech overlap, stop latency, correction uptake) and task completion. at the heart of the benchmark system is a simulated user that outputs frozen speech clips in order to drive the task-based interaction. two validations with human participants were conducted;: one to rate the frozen speech generation for fidelity and nanturalness of interaction, and one where real humans drive the interaction for a subset of the tasks, in place of the simultated user.

The core contribution is the combination of voice interaction metrics and task success in the same benchmark.

Critical strengths

the main strength of the paper is the novel aspect of one benchmark that combines both voice interaction aspects and task completion in one. the paper comes with extensive supplementary material detailing the experiments.

Critical weaknesses
  • The added value over existing benchmarks is not demonstrated. The novelty argument rests on a self-defined feature checklist (Table 1), not on evidence that combining full-duplex interaction with professional workflows reveals something existing benchmarks miss. No system is also evaluated on τ-Voice, DuplexWorld or Full-Duplex-Bench, so the paper cannot show that APEX-Voice ranks systems differently or exposes distinct failure modes. All reported metrics are the paper’s own, with no external reference point.

  • The contribution of full-duplex interaction is never isolated. Appendix B.4 states that the campaign does not vary modality condition. The voice-vs-text gap (Table 5) therefore confounds speech input/output, full-duplex dynamics, and the use of different, stronger models in the text condition. A matched turn-based voice condition, with the same tasks and corrections delivered as complete turns, is needed to support the claim that full-duplex interaction is a source of difficulty.

  • The benchmark only gives a partial account of the mechanics of full-duplex interaction, and misses out on several important phenomena:

  • Turn-taking is not measured, and may not be under model control. The floor-control metrics (Table 4) cover only overlap, barge-in yield and stop latency. There is no measure of response gap after the user’s end of turn, nor of the agent taking the floor during user pauses,
  • Backchannels are listed but not evaluated. Backchannels appear in the duplex-phenomenon taxonomy, but there is no metric for them in either direction and no count of how many workflows contain them; Figure 3 omits the duplex-phenomenon dimension. The qualitative appendix shows only scripted mid-utterance corrections as duplex events. Barge-in yield near 100% is presented as a strength, but if user backchannels are included in the barge-in set, yielding to them is an error, and a system that stops at any sound would score perfectly. Whether agents produce appropriate backchannels is not assessed.
  • No human reference makes the absolute numbers hard to interpret. Humans act only as simulated users (Study B), never as agents. Without a human-agent baseline it is unclear whether the ceiling is near 100%, and the text-agent results (about 60%) suggest it may not be. Task solvability was verified only by automated and LLM-judge checks.
Questions for authors

How do you explain the 1.2% PASS@1 of the cascaded baseline, given that text agents of comparable capability reach about 55%?

How are user turns endpointed for each system? Is end-of-turn detection performed by the harness or by the model? Can you report response gaps and false take-overs?

What disfluency phenomena does the user simulation produce in the audio, beyond fillers in the text? Does it include mid-turn pauses, hesitations, spontaneous overlap or backchannels? How many distinct TTS voices are used?

Can you report a human-agent baseline, e.g. on the 24-task audit subset, and per-system results for Study B?

Reviewer_hZXnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces APEX-Voice, a benchmark that tests full-duplex voice agents on 120 synthetic professional workflows. Each workflow ends in a work artifact (a record, request or plan) that can be checked.

  • The simulated user speaks from pre-generated audio: a flow engine picks a semantic plan, and one of several pre-recorded variants is played for it. Events such as mid-utterance corrections and barge-ins are scheduled on a shared media timeline.
  • Scoring: outcomes are graded by deterministic gates, including artifact validity (AV), plus an artifact field accuracy score (AFA). An LLM judge handles fields that aren't exact matches. Reported metrics include Pass@1, Reliable@3 and time-to-resolution.
  • Additional analyses: how agents handle mid-task corrections (Table 4), and a comparison of voice against text (Table 5).
  • Main finding: current voice agents complete only a small fraction of workflows.
Critical strengths
  1. A clear, well-structured evaluation design focused on corrections mid-trajectory. The benchmark targets a specific, practically important behaviour: whether a voice agent catches and applies a user's correction while a workflow is under way. Because corrections are scheduled on the media timeline and tracked down to individual artifact fields, failures can be attributed to the exact correction that was missed rather than inferred from overall success.
  2. Evaluation at several levels: process, actions and end state. Workflow success is decomposed into gates covering the process (e.g. required knowledge retrieval and approvals), the actions taken, and the validity of the final artifact. This gives a more complete picture than end-state checks alone.
    • Separating artifact validity (AV) from field-level accuracy (AFA) is especially useful. It distinguishes an agent that never produces a valid work product from one that produces it with a few wrong fields.
Critical weaknesses
  1. Validity of the benchmark itself is not established.

    • The tasks. The workflows are LLM-generated by the "APEX-VOICE Artifact Factory" (p.4) and filtered only by "deterministic as well as LLM-judge checks" (p.5). The paper doesn't show that they are realistic or representative professional workflows, or that their gold artifacts are correct. No expert or human review of tasks or gold answers is reported, and the relationship to the APEX methodology isn't explained beyond a citation.
    • The grading. Grading relies on a GPT-4o judge (p.6) with no reported calibration against human annotators.
    • The user simulation. App. F reports human ratings of utterance quality and a 24-task live-human check, but speech exists only "for every reachable benchmark-critical user plan" (p.15). It's unclear what happens when the agent asks a question that has no prepared answer, and the semantic-act observer that maps agent speech to user plans (p.6) isn't specified.

    Since every result depends on these three components, this limits how far the results can be trusted.

  2. The analyses characterize failures but don't isolate their causes.

    • Corrections (Table 4). There is no matched condition without corrections, so their effect can't be separated from task difficulty. The study also covers only mid-task corrections, a narrow subset of the communicative acts relevant to voice interaction.
    • Voice vs. text (Table 5). The text control isn't described, and text models are compared with different voice models; there's no same-model text baseline. It's also unclear whether the voice reference includes corrections.
    • Timing. The effect of barge-in timing isn't analysed, and neither is the difference between barging in on the agent and interjecting while it is silent.

    Overall, the characterization largely confirms expectations: agents struggle with long voice workflows and with corrections. The benchmark offers limited discrimination between systems; in Fig. 1, for example, time-to-resolution spans only about 30 s.

Other comments (presentation)

  • Table 1. The "Professional work" criterion is unclear: why is τ-Voice marked ✗ when its tasks also end in verifiable state changes?
  • The user is always cooperative. For example, negotiation tasks (Fig. 3; task 072, p.40) have no misaligned or adversarial counterpart, which limits the realism claim.
  • Fig. 1.
    • "Verifiable outcomes" should be defined where the figure is introduced.
    • The Pareto frontier is drawn only for Reliable@3, although Pass@1 is also plotted against time-to-resolution. App. H.1 explains why AFA vs. Pass@1 isn't used, but not why Pass@1 vs. time-to-resolution is excluded.
  • Table 2. Tool-use efficiency ("mean/oracle") needs a definition of the oracle.
  • Table 3. The last column is labelled "(AV)"; it should be AFA.
  • Eq. 2 and Eq. 5. The indicator function renders as "⊮"; use \mathbb{1}.
  • Typos: "valid artifact artifaction" (p.2); "user simulation simulation" (p.2 and the Table 1 caption).
Questions for authors
  1. (Clarification) Were any tasks or gold artifacts reviewed by domain experts or humans? If so, how many, and what was the agreement?
  2. (Clarification) Has the GPT-4o judge been validated against human graders? What is the agreement?
  3. (Clarification) What happens when the agent asks a question that has no matching user plan, and how often does this occur? How does the semantic-act observer work?
  4. (Clarification) How is the text control in Table 5 implemented? Can you report a same-model text vs. voice comparison?
  5. (Clarification) Can you report results on matched tasks with and without corrections?
  6. (Future work) Does barge-in timing, or interrupting the agent vs. speaking while it is silent, affect success? Could you add misaligned or adversarial users, for example in negotiations?
  7. (Clarification) How does APEX-Voice's task construction relate to the APEX methodology?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.