- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces DuplexAct, a bilingual benchmark for evaluating natural and controllable interaction behaviors in real-time full-duplex speech systems. It covers six interaction families across two tracks: Capability, which evaluates context-appropriate behaviors, and Command, which assesses adherence to user-spoken instructions. Experiments on seven systems reveal substantial limitations in complex turn-taking and instruction-conditioned interaction.
Critical strengths
-
Beyond timing-based turn-taking. The paper extends turn-taking evaluation beyond simple timing prediction to context- and policy-conditioned interaction decisions, addressing an important aspect of natural spoken dialogue.
-
Opposing-policy evaluation. Evaluating the same conversational input under opposing user instructions is a valuable design for assessing whether models can genuinely adapt their interaction behaviors to user preferences.
Critical weaknesses
-
Limited distinction between Capability and Command. In several interaction families (e.g., Silence, Wait-Completion, and Immediate), the Command track largely reinforces behaviors that models should already perform without explicit instructions. This weakens its ability to evaluate genuine policy-conditioned behavioral adaptation.
-
Unclear role of the T × I × R ontology. The benchmark largely relies on predefined combinations of temporal configurations, interactional interpretations, and expected responses. It is unclear how the proposed formulation systematically guides benchmark construction or disentangles these dimensions.
-
Insufficient data construction details. Important information is missing regarding the sampling and selection of real conversations, LLM-assisted rewriting procedures, and data filtering criteria. These omissions limit the reproducibility of the dataset construction process.
-
Insufficient diversity analysis. More analysis is needed to establish whether the benchmark preserves sufficient linguistic, temporal, interactional, and acoustic diversity. The reported human validation primarily supports label correctness rather than dataset diversity or representativeness.
-
Limited coverage of backchannel behaviors. While the benchmark evaluates how systems respond to user backchannels, it does not explicitly assess the ability to proactively generate appropriate backchannels, an important aspect of natural full-duplex interaction.
-
Difficult-to-follow presentation. The paper introduces numerous abstract concepts and complex formulations without sufficient intuitive explanations. Important details about the execution flows and scoring procedures are scattered across the main text and appendix, making the benchmark design unnecessarily difficult to understand.
Overall, I find the target problem of the paper interesting, but the technical quality and presentation make it unsuitable for publication for now. I think the paper needs substantial revision before publication.
Questions for authors
N/A
2 comments
Thank the authors for their rebuttal. I will maintain my current assessment.
Thank you for the careful review.
Q1. Capability and Command
Capability tests context-appropriate behavior under a fixed base policy. Command tests adherence to policies the user states or updates through speech. A user can reinforce the base policy, so both tracks can require the same action (§3.1).
In Command, Immediate requires holding until the question ends, then answering promptly. Silence and Wait-Completion require waiting until explicit release, then responding. A wait-for-release variant of Immediate would overlap with these tasks (Appendix C.1, C.3–C.5). Affective-Adaptation pairs the same probe with instructions to INTERVENE during the affective passage or to HOLD until the user finishes and then respond (C.1, C.7). In one Barge-In-Yield session, the same backchannel requires CONTINUE by default, YIELD after activation, and CONTINUE after release (C.9, Table 9). These conditions test both execution under spoken instructions and adaptation when a policy change requires a different action.
Q2. Role of the T × I × R ontology
Lu et al. (2026) describe interactions by temporal relation (T), user intent (I), and system response (R). We use I for the interpretation of events such as errors or noise and make policy explicit. Together, timing, interpretation, and policy determine the required action (§3.1).
This guides scenario selection, case construction, and target actions. The families test when to wait or respond (Silence, Wait-Completion, Immediate), whether to intervene during user speech (Correction, Affective-Adaptation), and whether to yield or continue during overlap (Barge-In-Yield). We retain context and decision anchors: a pause can mark an unfinished utterance (HOLD) or a completed request (RESPOND). Under the base policy, overlap can signal a floor claim (YIELD) or a backchannel (CONTINUE) (§§3.1–3.2, Table 1).
Q3. Data construction
We built self-contained cases from customer-service records authorized for research, preserving utterances, context, timing, and decision anchors. We de-identified scenarios before speech production. We use no original customer or agent audio (§3.2).
LLMs helped adapt de-identified records, localize cases into English, and draft Barge passages using Wikipedia. We preserved roles, numerical and logical relations, target errors, and interaction structure. All AI-assisted content was manually reviewed and corrected as needed. C.8 documents the passages and ESC-50 noise.
All three annotators had to independently approve the policy-specific label and recording (pronunciation, prosody, pauses, fluency, and intended emotion where applicable). On 120 sampled items, with reference actions and subsequent audio withheld, 97.8% of Capability and 99.4% of Command judgments matched the reference (§3.2, Appendix D, Table 10). Training and authorization records require clearance and de-identification. After dataset release, we will share the authorization documents cleared for disclosure.
Q4. Diversity analysis
DeepSeek (deepseek-v4-pro, temperature = 0) extracted each user’s goal, issue state, and desired outcome from the utterance, history, and available background. It rated all within-family pairs: 0 = unrelated, 1 = different requests in one domain, 2 = same request in different scenarios, 3 = same scenario with different key conditions, 4 = near-duplicates differing only in wording or nonessential slot values. For 3s and 4s, the same model assigned final ratings from the full available text with initial scores hidden.
Each Chinese source case counted once across translations, phrasings, and repeated events. We audited 48 Barge passages separately (1,128 pairs) and will report the audit in the final paper.
Q5. Backchannel coverage
Our six families cover a subset of natural full-duplex behaviors. Barge-In-Yield tests whether the system distinguishes a user backchannel from an interruption and follows the active policy (§3.1, C.8–C.9). We will clarify this scope. Future work includes evaluating the timing, content, and effect of system-generated backchannels on user speech.
Q6. Presentation
Figure 1 illustrates the families. §3.1 defines the tracks and actions (Table 1). §3.3 describes real-time input and actual audio playout. Table 7 (C.1) shows session structures. Table 9 (C.9) traces eight Barge Command events. §3.4 and Table 2 define scoring and aggregation. Appendix H reports response latency for successful Immediate and Wait-Completion units. Table 11 (E.2) covers boundary cases. Appendix G gives a case for each family and track.