acceptodds
Under review as a conference paper at ICLR 2027

DuplexAct: Benchmarking Natural and Controllable Interaction Behaviors in Real-Time Full-Duplex Speech Systems

19 views

Abstract

Real-time full-duplex speech systems can listen and speak simultaneously, but natural interaction also requires them to adapt their behavior to the conversational context and users’ spoken instructions. We introduce DuplexAct, a benchmark for evaluating how naturally and controllably these systems interact. DuplexAct covers six interaction families, each evaluated on two tracks: Capability and Command. The Capability track tests whether a system acts appropriately in the conversational context under a fixed policy for each family. The Command track evaluates whether a system follows users’ spoken instructions and adapts its behavior when those instructions are updated. We build 512 base cases in Chinese and English from real customer-service conversation records and validate the resulting dataset through human review. Our real-time evaluation protocol accommodates different system interfaces, assesses interaction behavior from actual audio playout traces, and reports correctness and latency separately. Most evaluated systems exceed 90% success in initiating responses to completed requests in both languages under the Capability track. In contrast, success rates reach at most 32.1% for waiting through a mid-utterance pause before responding, and 30.8% for timely intervention after factual errors. These behaviors remain challenging in the Command track. These findings highlight context-appropriate behavior and reliable spoken-policy execution as key challenges for natural and controllable full-duplex interaction.

Primary area:
datasets and benchmarks
Keywords:
Full-duplex speech systems, Benchmark and evaluation, Spoken instruction following, Turn-taking, Interaction controllability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eByXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DuplexAct, a bilingual benchmark for evaluating natural and controllable interaction behaviors in real-time full-duplex speech systems. It covers six interaction families across two tracks: Capability, which evaluates context-appropriate behaviors, and Command, which assesses adherence to user-spoken instructions. Experiments on seven systems reveal substantial limitations in complex turn-taking and instruction-conditioned interaction.

Critical strengths
  1. Beyond timing-based turn-taking. The paper extends turn-taking evaluation beyond simple timing prediction to context- and policy-conditioned interaction decisions, addressing an important aspect of natural spoken dialogue.

  2. Opposing-policy evaluation. Evaluating the same conversational input under opposing user instructions is a valuable design for assessing whether models can genuinely adapt their interaction behaviors to user preferences.

Critical weaknesses
  1. Limited distinction between Capability and Command. In several interaction families (e.g., Silence, Wait-Completion, and Immediate), the Command track largely reinforces behaviors that models should already perform without explicit instructions. This weakens its ability to evaluate genuine policy-conditioned behavioral adaptation.

  2. Unclear role of the T × I × R ontology. The benchmark largely relies on predefined combinations of temporal configurations, interactional interpretations, and expected responses. It is unclear how the proposed formulation systematically guides benchmark construction or disentangles these dimensions.

  3. Insufficient data construction details. Important information is missing regarding the sampling and selection of real conversations, LLM-assisted rewriting procedures, and data filtering criteria. These omissions limit the reproducibility of the dataset construction process.

  4. Insufficient diversity analysis. More analysis is needed to establish whether the benchmark preserves sufficient linguistic, temporal, interactional, and acoustic diversity. The reported human validation primarily supports label correctness rather than dataset diversity or representativeness.

  5. Limited coverage of backchannel behaviors. While the benchmark evaluates how systems respond to user backchannels, it does not explicitly assess the ability to proactively generate appropriate backchannels, an important aspect of natural full-duplex interaction.

  6. Difficult-to-follow presentation. The paper introduces numerous abstract concepts and complex formulations without sufficient intuitive explanations. Important details about the execution flows and scoring procedures are scattered across the main text and appendix, making the benchmark design unnecessarily difficult to understand.

Overall, I find the target problem of the paper interesting, but the technical quality and presentation make it unsuitable for publication for now. I think the paper needs substantial revision before publication.

Questions for authors

N/A

2 comments

Thank you for the careful review.

Q1. Capability and Command

Capability tests context-appropriate behavior under a fixed base policy. Command tests adherence to policies the user states or updates through speech. A user can reinforce the base policy, so both tracks can require the same action (§3.1).

In Command, Immediate requires holding until the question ends, then answering promptly. Silence and Wait-Completion require waiting until explicit release, then responding. A wait-for-release variant of Immediate would overlap with these tasks (Appendix C.1, C.3–C.5). Affective-Adaptation pairs the same probe with instructions to INTERVENE during the affective passage or to HOLD until the user finishes and then respond (C.1, C.7). In one Barge-In-Yield session, the same backchannel requires CONTINUE by default, YIELD after activation, and CONTINUE after release (C.9, Table 9). These conditions test both execution under spoken instructions and adaptation when a policy change requires a different action.

Q2. Role of the T × I × R ontology

Lu et al. (2026) describe interactions by temporal relation (T), user intent (I), and system response (R). We use I for the interpretation of events such as errors or noise and make policy explicit. Together, timing, interpretation, and policy determine the required action (§3.1).

This guides scenario selection, case construction, and target actions. The families test when to wait or respond (Silence, Wait-Completion, Immediate), whether to intervene during user speech (Correction, Affective-Adaptation), and whether to yield or continue during overlap (Barge-In-Yield). We retain context and decision anchors: a pause can mark an unfinished utterance (HOLD) or a completed request (RESPOND). Under the base policy, overlap can signal a floor claim (YIELD) or a backchannel (CONTINUE) (§§3.1–3.2, Table 1).

Q3. Data construction

We built self-contained cases from customer-service records authorized for research, preserving utterances, context, timing, and decision anchors. We de-identified scenarios before speech production. We use no original customer or agent audio (§3.2).

LLMs helped adapt de-identified records, localize cases into English, and draft Barge passages using Wikipedia. We preserved roles, numerical and logical relations, target errors, and interaction structure. All AI-assisted content was manually reviewed and corrected as needed. C.8 documents the passages and ESC-50 noise.

All three annotators had to independently approve the policy-specific label and recording (pronunciation, prosody, pauses, fluency, and intended emotion where applicable). On 120 sampled items, with reference actions and subsequent audio withheld, 97.8% of Capability and 99.4% of Command judgments matched the reference (§3.2, Appendix D, Table 10). Training and authorization records require clearance and de-identification. After dataset release, we will share the authorization documents cleared for disclosure.

Q4. Diversity analysis

DeepSeek (deepseek-v4-pro, temperature = 0) extracted each user’s goal, issue state, and desired outcome from the utterance, history, and available background. It rated all within-family pairs: 0 = unrelated, 1 = different requests in one domain, 2 = same request in different scenarios, 3 = same scenario with different key conditions, 4 = near-duplicates differing only in wording or nonessential slot values. For 3s and 4s, the same model assigned final ratings from the full available text with initial scores hidden.

FamilySource casesPairsCases in ≥1 near-duplicate pair
Silence571,5964 (7.02%)
Wait-Completion283780 (0.00%)
Immediate843,4864 (4.76%)
Correction397410 (0.00%)
Affective-Adaptation181532 (11.11%)
Total2266,35410 (4.42%)

Each Chinese source case counted once across translations, phrasings, and repeated events. We audited 48 Barge passages separately (1,128 pairs) and will report the audit in the final paper.

Q5. Backchannel coverage

Our six families cover a subset of natural full-duplex behaviors. Barge-In-Yield tests whether the system distinguishes a user backchannel from an interruption and follows the active policy (§3.1, C.8–C.9). We will clarify this scope. Future work includes evaluating the timing, content, and effect of system-generated backchannels on user speech.

Q6. Presentation

Figure 1 illustrates the families. §3.1 defines the tracks and actions (Table 1). §3.3 describes real-time input and actual audio playout. Table 7 (C.1) shows session structures. Table 9 (C.9) traces eight Barge Command events. §3.4 and Table 2 define scoring and aggregation. Appendix H reports response latency for successful Immediate and Wait-Completion units. Table 11 (E.2) covers boundary cases. Appendix G gives a case for each family and track.

Thank the authors for their rebuttal. I will maintain my current assessment.

Reviewer_saGFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes a new benchmark, DuplexAct, designed for the evaluation of naturalness and controllability in spoken dialogue system interactions. It includes 512 interaction cases (256 in Chinese and 256 in English) from real customer-service conversation records and is validated using human review. The key contribution is the benchmark itself, which is extensive and seems like a useful resource.

Critical strengths

This paper was well-written and I think the resulting resource is high-quality and will be useful. The related work section of the paper was also sufficiently extensive to provide a nice background of the problem and the prior work in this area. The results establish strong benchmarks upon which others can build.

Critical weaknesses

The paper could benefit from more details about the annotators and data sources---for instance, it was unclear whether the customer service conversations all took place within a relatively narrowly scoped domain, or across multiple domains. If the former, it seems like it may be hard to draw many generalizable conclusions from experiments with this benchmark. Additionally, it was unclear how much "real" content is ultimately in the dataset. For instance, the paper states that "Barge-In-Yield uses manually reviewed, LLM-assisted reading materials based on public Wikipedia information," which was confusing because it seems like this then wasn't part of the customer service interaction at all. Also, in the AI Use Statement it becomes clear that the English scenarios were created using machine translation, and in the Ethics Statement it becomes clear that native speakers recorded the Chinese scenarios but I guess the English scenarios must have used generated speech. Both of these differences could introduce large variations in performance across the two languages.

Questions for authors

A few things were unclear to me (these aren't huge, but it'd be nice to update them in the revised manuscript):

  • "Each family scores the complete behavioral contract of its policy" -> what does this mean?
  • I'm not sure if I understand how the full evaluation of one system takes 5,114 planned sessions. What exactly is a planned session, and from where does this number come?
  • It'd be helpful to provide more details about the annotators in the main text.

Some larger questions:

  • How much of the final dataset is actual human data versus machine-generated data (and when I say machine-generated, I guess I mean machine translations, machine-generated acoustic output, or anything that wasn't directly recorded from a real human setting)?
  • There seems to be quite a bit of performance variation between Chinese and English settings. Any thoughts on why? Is this perhaps due to differences in recording/data collection setting (it seems like perhaps the English content required more synthetic processes)? Do these performance differences weaken the benchmark's ability to lead to generalizable performance claims?
3 comments

Thank you for your thoughtful review and positive assessment.

Q1. Service domains

The calls cover retail refunds, parcel delivery, airline ticket changes and cancellations, insurance inquiries, utility payments, automatic subscription renewals, and account security, from initial inquiries to billing disputes. Decisions such as waiting through a pause or yielding to an interruption depend primarily on speech timing and interactional function rather than domain knowledge. In Correction, the system receives the facts needed to identify the error (Appendix B.4). We will add domain examples to §3.2 and discuss the limits of generalizing beyond customer service in §5.

Q2. Data sources and production

Text. The 256 Chinese base cases derive from real customer-service calls authorized for research. LLMs helped rewrite de-identified events into self-contained cases and translate and localize them into 256 English counterparts. Human reviewers checked and corrected all AI-assisted content (§3.2, AI Use Statement). The only text not taken from the calls is the passage that the system reads aloud in Barge-In-Yield.

Audio. Six native Mandarin speakers recorded 18 Chinese Affective-Adaptation cases (3.5% of 512). The other 494 used synthetic speech: emotion-controlled CosyVoice 3.5 for English Affective and voice cloning with anonymized voices elsewhere (§3.2, Ethics Statement). All 960 spoken Command instructions were synthesized. Barge uses ESC-50 noise recordings (Appendix C.8). We retained an item only if all three annotators approved both its label and its recording. Across both languages, each system requires 5,114 planned sessions.

Barge-In-Yield. Fixed Wikipedia-based reading passages reduce variation in content and duration and keep the system speaking so a user stimulus can overlap its speech at a planned point. The passages were LLM-assisted and human-reviewed (Appendix C.8). We will bring the sources, production methods, review process, and rationale together in §3.2.

Q3. Cross-language comparability

English scenarios are localized from Chinese. Affective speech is human-recorded in Chinese and synthesized in English. All other families use synthesized speech in both languages. Affective Command pairs the same audio with opposing instructions within each language, allowing us to test whether the system intervenes or keeps listening as instructed. This tests instruction following within each language without relying on cross-language score comparisons.

Production differences may affect individual scores, so we report results separately by language (§3.4). In both languages, five systems exceed 90% on Immediate Capability, and none exceeds 32.1% on Wait-Completion Capability (Table 3). We will discuss cross-language differences in §4.2.

Q4. Complete behavioral contracts

The contract covers every action and constraint required by the policy (§3.4, Table 2). For tasks scored under Eq. 3, all required decisions must pass. In Wait-Completion Capability, a system responded 1.76 s after the user finished but gave advice too early at the separately tested pause. The case therefore failed (Appendix G.2). We will clarify this meaning where the phrase first appears.

Q5. Planned sessions

A planned session is one protocol-defined real-time run in which we stream input audio and record system output. One base case can require several sessions to test different decisions, wordings, or policies. Wait-Completion Capability tests the incomplete utterance, pause, and completion separately. Affective Command tests two opposing policies for each of five instruction variants. Sessions from the same base case are not independent samples.

Per language, 256 base cases yield 372 Capability and 2,185 Command sessions. Across Chinese and English, this gives 2 × (372 + 2,185) = 5,114 planned sessions per system. Table 7 (Appendix C.1) gives the execution structure, and Table 8 (Appendix C.2) gives the counts. We will explain the total and cite Table 8 in §1.

Q6. Annotators and validation

The annotators were customer-service professionals at a large financial technology company. Their hotline work involved identifying callers' needs, managing turns, and handling emotionally charged calls. We will add their backgrounds to §3.2.

Item review. Three annotators independently checked each interaction label under the applicable policy. For the recording, they assessed pronunciation, prosody, pause placement, fluency, and, for affective items, the intended emotion. We retained an item only if all three approved both the label and the recording (§3.2).

Blind validation. Three annotators independently judged the required action at 120 sampled decision points with reference labels and subsequent audio withheld. Agreement with the reference action was 97.8% for Capability and 99.4% for Command, counting each individual judgment and treating uncertain responses as mismatches (Appendix D).

Thanks for the response! This is helpful. I think for this paper to be ready for acceptance, the details regarding data sources and production (and language-specific differences in data sources/production) need to be moved into the main text body and explained more, since having that understanding is essential for fully evaluating the work.

Thank you for your helpful feedback. We have added the requested details to Sections 3.2, 3.3, and 4.2 of the manuscript and uploaded the revised PDF. We hope these changes address your concern and would be grateful if you would reconsider your score.

Reviewer_Fc1POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DuplexAct, a bilingual benchmark for evaluating interaction behavior in real-time full-duplex speech systems, comprising 512 base cases across six interaction families and two tracks: Capability, which tests behavior under fixed policies, and Command, which tests adherence to spoken instructions. The paper propose joint behavioral requirements, matched opposing-policy conditions, and evaluation based on actual audio playout, focusing on action and timing compliance. Experiments on seven systems show that reliable response initiation does not imply reliable waiting, intervention, or policy control, while human validation supports the reference actions and decision-level analysis reveals specific behavioral bottlenecks.

Critical strengths
  • The proposed evaluation framework is valuable. Joint scoring reveals behavioral weaknesses that previous benchmarks may overlook, while applying opposing spoken policies to the same input helps distinguish instruction adherence from fixed response tendencies.
  • The focused problem is important and the paper offers useful insights for the design of future systems.
Critical weaknesses
  • Some scenarios contain too few base cases. Although instruction variations generate more test instances, they do not increase the diversity of the underlying scenarios, which may weaken the reliability of the conclusions.
  • The choice of evaluation windows is insufficiently discussed. Why were the 6-second and 3-second response windows selected? An ablation study varying the window lengths would help justify these choices and assess the robustness of the results.
Questions for authors
  • Can you provide confidence intervals clustered by base case and repeated-run results for the main comparisons, particularly for the smaller task families?
  • Presenting key test examples more prominently in the main text would help readers understand the paper’s core design and contributions.
2 comments

Thank you for recognizing the value of joint scoring and opposing-policy evaluation.

Q1. Base-case counts and reliability of the conclusions

The data support our main finding: timely responses or successful intervention do not imply reliable policy-conditioned action selection.

We computed confidence intervals using 10,000 bootstrap resamples clustered by base case. All instruction variants derived from the same base case are resampled together rather than treated as independent observations. The table below presents the two examples highlighted in the Introduction. We will include the full confidence-interval results in an appendix to the final paper and add a reference to them in §3.4 and the caption of Table 3.

Model / languageTask / trackBase casesSuccess (95% CI)
Qwen Audio / ENImmediate / Capability8495.2% [90.5%, 98.8%]
Qwen Audio / ENWait-Completion / Capability2832.1% [14.3%, 50.0%]
MiniCPM / ZHAffective / Capability18100% [100%, 100%]
MiniCPM / ZHAffective / Command1810.0% [4.4%, 15.6%]

Q2. Evaluation windows and sensitivity analysis

Based on listening assessments and customer-service expert input, we allow 6 seconds for response onset after request completion or constraint release, and 3 seconds for intervention and yielding, which require faster action. Windows begin at task-specific anchors (§3.4, Table 2).

Related work provides context: without fillers, a 6.5-second delay reduced willingness to interact again compared with 1.5 seconds [1]; TurnBench uses a 3-second deadline for end-of-turn and interruption detection [2].

We fixed MiniCPM's outputs and other scoring conditions, then rescored Command results under different deadlines. Cells report Chinese / English success rates.

Silence: Response deadline

Metric5 s6 s (default)7 s
Command success29.5% / 14.7%34.0% / 25.3%34.0% / 30.5%

Barge-In-Yield: Stopping deadline

Metric2 s3 s (default)4 s
Command success53.1% / 46.7%53.2% / 46.7%53.3% / 46.7%
CONTINUE87.6% / 73.8%87.6% / 73.8%87.6% / 73.8%
YIELD1.5% / 6.2%1.7% / 6.2%1.9% / 6.2%

For Barge-In-Yield, the 1.5-second post-stop stability requirement and events (1,192 Chinese; 600 English) remained fixed. Longer windows improve some scores, but Silence Command success remains low, and yielding remains much less successful than continuing. These findings hold across the tested ranges. We will clarify the rationale in §3.4 and add this analysis to an appendix.

Q3. Consistency of results across runs

We examined the historical Barge-In-Yield MAIN/R1 comparison: four models, each with 30 paired sessions (6 base cases × 5 instruction phrasings). Assets and main model configurations were unchanged; R1 varied policy order and event timing. This tests consistency under these variations, rather than identical-condition repeatability.

We re-extracted evidence from actual audio playout using the same scoring rules. Only matched events technically valid and unambiguously scorable in both runs are included. Agreement means the same success/failure outcome.

ModelSuccess: MAIN → R1Agreement
GPT Realtime 247.4% → 46.9%96.7%
Qwen3.5 Omni Plus Realtime39.3% → 39.8%95.5%
Qwen Audio 3.0 Realtime Plus62.1% → 57.8%93.8%
Gemini 3.1 Flash Live41.2% → 37.6%90.4%

The first three models show more consistent outcomes. Gemini's overall score changes little, but many individual outcomes differ.

We also bounded success rates including unresolved events. Among Qwen Audio's 240 paired events technically valid in both runs, counting unresolved CONTINUE events as failures gives lower bounds of 86.1% / 83.3% (MAIN/R1). Counting unresolved YIELD events as successes gives upper bounds of only 15.6% / 11.5%. The model can continue speaking but struggles to yield as instructed in both runs, despite changes in policy order and timing. We will provide settings, paired-event counts, and unresolved-event handling in the supplement.

Q4. Presentation of key test cases in the main text

Figure 1 illustrates the six families; Table 2 specifies their scoring criteria. In Appendix G.2, a system responds 1.76 seconds after completion but speaks prematurely at the independently executed pause node, so the case fails. Checking only the completion node would miss this violation. Appendix C.9 and Table 9 show the same backchannel requiring CONTINUE → YIELD → CONTINUE as policy is activated and released. We will add these examples to §3.4 and §3.1, respectively, with appendix references.

[1] Maslych et al. (2025). Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents. CUI.

[2] Jiang et al. (2026). TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue. SLT.

Thank you for running the additional experiments. I will keep my score unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF and summarize the main evidence and revisions below.

  1. Reliability (Fc1P). In our responses, we report base-case-clustered 95% confidence intervals: Qwen Audio scores 95.2% [90.5%, 98.8%] on English Immediate Capability, compared with 32.1% [14.3%, 50.0%] on Wait-Completion. This supports the distinction between timely response initiation and reliable waiting through an unfinished turn. Timing-window sensitivity tests and a four-model Barge comparison provide further evidence under the tested conditions.
  2. Data and validation (saGF, eByX). Revised §3.2 consolidates data provenance, language-specific production methods, and annotator backgrounds. Blind validation yielded 97.8% and 99.4% human–reference action agreement for Capability and Command, supporting the reference actions at the sampled decision points. Our response also reports a semantic audit in which 10 of 226 source cases (4.42%) appeared in at least one near-duplicate pair, suggesting limited semantic redundancy among the audited cases.
  3. Evaluation design and presentation (all reviewers). We clarify the two evaluation tracks and add concrete examples of policy-dependent actions and joint scoring (§§3.1 and 3.4). We also explain the 5,114 planned sessions per system, distinguishing them from independent samples (§3.3), and clarify the limits of cross-language interpretation and cross-domain generalization (§§4.2 and 5). Detailed evidence and qualifications are provided in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.