acceptodds
Under review as a conference paper at ICLR 2027

DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

Abstract

End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriate interaction actions as conversations unfold. However, existing benchmarks lack a structured action-level formulation, often evaluate action occurrence rather than success, and provide limited coverage of diverse real-world conversations. We introduce DuplexAct, a benchmark for evaluating interaction action taking in E2E FD-SLMs. DuplexAct defines a three-level action taxonomy spanning eight interaction actions and 24 corresponding interaction conditions. We further characterize successful action-taking through immediate execution and subsequent follow-through, and develop action-specific evaluation metrics for both. DuplexAct contains 4,691 test instances spanning casual conversation, meetings, customer service, and interviews, including 2,531 instances derived from real-world conversational recordings. Together, DuplexAct provides a structured and real-world-grounded framework for systematically assessing how E2E FD-SLMs participate in ongoing spoken interaction.

Primary area:
datasets and benchmarks
Keywords:
Full-duplex, End-to-end spoken language models, Action-taking, Benchmarking and evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WDVaOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+2)
Read review
Paper summary

The paper points out that existing benchmarks for end-to-end full-duplex spoken language models (FD-SLMs) lack structured action-level definitions and overly focus on surface occurrences rather than pragmatic success. To address this, the authors propose DUPLEXACT, a benchmark grounded in human Interactional Competence, featuring 8 actions and 24 conditions. This benchmark strictly divides action evaluation into two independent stages: "immediate execution" and "subsequent follow-through". The dataset contains 4,691 instances, combining real-world corpora and controlled synthesized data. Evaluations of 14 frontier models reveal a systemic deficiency in understanding the deeper semantic consequences of interactive actions.

Critical strengths
  • Evaluation Dimensions: The paper extends evaluation beyond metrics such as response latency and takeover rates by introducing the “subsequent follow-through” metric. This metric captures cases where models interrupt at an appropriate time but do not provide an appropriate response afterward.
  • Taxonomy: The classification of 8 actions and 24 conditions, based on Interactional Competence, provides a comprehensive framework for studying full-duplex dialogue control.
Critical weaknesses
  • Validation of Automated Metrics: The success of “subsequent follow-through” is evaluated using an LLM judge, including whether the model simplified the language, revised obsolete content, or shifted the topic. The paper reports human annotator agreement (97.9% and 95.5%) during dataset construction but does not report agreement between the LLM judge’s scores and human judgments. The reliability of the automated evaluation therefore remains unclear.
  • Evaluation Pipeline Requirements: Simulator Replay (covering Stop, Hold Floor, Recover, and Interrupt) requires streaming text transcripts aligned with audio progress. This requirement prevented the evaluation of 4 of the 14 tested open-source models. It may also limit the benchmark’s applicability to audio-to-audio models that do not produce text transcripts.
  • Acoustic Properties of Synthesized Data: For L2 conditions involving high cognitive load, such as “Safety,” the paper uses an LLM-to-TTS synthesis pipeline. Emergency interruptions can involve changes in speech rate and pitch, as well as overlapping speech. Whether the synthesized data reproduce these acoustic features is unclear, leaving the ecological validity of these conditions unverified.
Questions for authors
  1. As the core innovation of this benchmark, the "subsequent follow-through" evaluation relies heavily on an LLM judge's assessment of pragmatic consistency. Did the authors conduct any Human-LLM Agreement validation (e.g., Cohen's Kappa) on a subset of the test set? Without this validation, how can the community trust that the benchmark scores accurately reflect true interactive performance?
  2. Given that Simulator Replay heavily relies on streaming text transcripts, how can the authors prove that DUPLEXACT will be able to adapt to future pure audio-to-audio E2E models that discard intermediate text representations? Have you considered trigger alternatives based on low-latency VAD or acoustic features?
  3. Regarding the TTS-synthesized interruption data for "Safety" and "Help" scenarios, how do the authors ensure that the generated speech contains the unique paralinguistic and prosodic features typical of humans facing danger or needing urgent assistance? Does this lead to models merely overfitting to specific text semantics rather than learning authentic acoustic interaction dynamics?
4 comments

We thank the reviewer for recognizing the value of our Subsequent follow-through evaluation in revealing limitations overlooked by existing metrics, particularly models' failure to fulfill the intended purpose of their actions. We also appreciate reviewer’s positive assessment of our structured, Interactional Competence-grounded taxonomy, which provides a systematic framework for evaluating full-duplex interaction capabilities.

The revised and newly added content is marked in red in the PDF for convenience.


Weakness 1 and Question 1: Subsequent follow-through relies on LLM judgments without human–LLM agreement validation, raising concerns about the reliability of benchmark scores.

Response to Weakness 1 and Question 1: We thank the reviewer for raising this concern. We sampled 454 instances from the seven actions (Stop is not included, which uses rule-based criteria) that require LLM judgment. YES and NO cases were balanced (217 vs. 237) so that agreement is not inflated by the more common label. Ten annotators were recruited, and each instance was labeled independently by two of them. Annotators saw the same input and instructions as GPT-5.4. For the 115 instances where the two disagreed, a third annotator gave a label, and the majority vote became the human reference.

The validation results are shown in Table R7. GPT-5.4 agrees with the human reference on 77.1% of instances. This is on par with the 74.7% agreement between two human annotators. A judge cannot be expected to agree with humans more than humans agree with each other. The remaining disagreement reflects how subjective it is to judge pragmatic follow-through, rather than a weakness specific to GPT-5.4.

Table R7. Human–human agreement and GPT‑5.4 agreement with the human reference across 454 instances of seven actions.

ActionHuman–Human AgreementGPT–Human Reference Agreement
Wait75.0%82.1%
Ignore72.0%82.9%
Take Turn88.1%72.6%
Backchannel72.7%77.3%
Hold Floor66.7%75.0%
Recover63.3%71.7%
Interrupt82.5%75.0%
Overall74.7%77.1%

Weakness 2 and Question 2: Simulator Replay requires intermediate streaming text representation, limiting model coverage and raising concerns about compatibility with future pure audio-to-audio models. Could low-latency VAD or acoustic features serve as alternative triggers?

Response to Weakness 2 and Question 2: The key contributions of our work is the action taxonomy, success criteria, and benchmark data coverage, which are applicable to pure audio-to-audio models. The reliance on intermediate streaming transcripts is a deliberate choice in the current simulator implementation.

Simulator Replay must inject recorded user speech at contextually appropriate points in the model's ongoing response (e.g., Recover and Hold Floor). Since different models generate different content, these points cannot be fixed in advance and require semantic understanding of the ongoing response. Following τ-Voice, we adopt an LLM-driven simulator using intermediate streaming transcripts to make low-latency interaction decisions. VAD or acoustic features alone cannot reliably identify these semantically appropriate intervention points.

Applying external streaming ASR to the model's output audio is a possible alternative, but its additional latency and transcription accuracy would further shift the injection points and compromise comparability across models. Therefore, the resilience on the streaming text transcripts is currently the most practical choice.

We agree that full-duplex models may move toward pure audio-to-audio processing, and the simulator can evolve in the same direction by taking the model's output audio directly as input with better semantic understanding, low-latency decision-making, and more controllable speech output content. As they mature, the text-based module can be replaced without affecting the benchmark design, since the contributions are independent of the simulator's input modality.

Weakness 3 and Question 3: TTS-generated Safety and Help scenarios may lack realistic prosodic and paralinguistic cues, raising concerns about ecological validity and overreliance on text semantics.

Response to Weakness 3 and Question 3: Our Safety and Help conditions target a different case: users who are unaware of a risk or of their need for help. In these cases, there is no urgency or distress in the user's voice to detect. The model must infer the need to intervene from what the user is saying. For instance:

  • Safety: A user calmly mentions, "I found a small mercury thermometer broken in the nursery dresser, and I'm vacuuming up the shiny beads from the carpet..." Unaware of the danger, the user continues speaking, and the model should proactively interrupt to warn against vacuuming mercury.
  • Help: A user may calmly discuss several travel options, going back and forth between them without showing the hesitant tones to ask for help. The model should step in with a suggestion.

Distress prosody is therefore absent by design, not a limitation of TTS. Relying on content in these cases is not overfitting to text semantics. Instead, recognizing a risk or a need for help that is not acoustically signaled is precisely the capability these conditions evaluate (Table 7).

Reviewer_WDVaOpenReview ↗

Response to Authors

Dear authors,

Thank you for the substantial additional work during the rebuttal period and your clarifications on the streaming-transcript requirement and the design of the Safety and Help scenarios. Since full-duplex dialogue involves human–machine interaction, my main concern lies on whether the LLM score can truly reflect human recognition of this benchmark score. The human alignment experiment supplemented by the authors have addressed my concerns. I will update my review and score accordingly.

Dear Reviewer,

Thank you for your positive feedback and for acknowledging our additional efforts. We sincerely appreciate your thoughtful comments and your willingness to reconsider and increase the score.

Reviewer_kA7cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DuplexAct evaluates interaction control in full-duplex spoken language models using a hierarchy of participation state, eight interaction actions, and 24 action conditions. The benchmark contains 4,691 instances across four conversational settings, including 2,531 instances derived from real recordings. It distinguishes immediate execution from subsequent follow-through and uses direct audio replay for four actions and simulator-controlled interaction for four others. Fourteen models are evaluated on direct replay and ten on simulator-based actions. The reported results show uneven action coverage and frequent failures to follow through after an action has been executed. The appendix specifies timing rules, judging prompts, data-label human validation, and condition-level results.

Critical strengths
  1. Separating what the model should do from the situation in which it should do it makes coverage easier to inspect. Evaluating whether a response actually incorporates an interruption or fulfills a backchannel's function is more informative than counting an action's occurrence. This is a meaningful direction for full-duplex evaluation.

  2. It covers eight actions across four settings and includes 2,531 instances from natural conversations. Appendix A.4 reports five human judgments per sampled instance for 468 instances, and Appendix B supplies operational criteria and prompts. Evaluation across both open and closed systems, with condition-level counts in Table 16, makes the benchmark potentially valuable for diagnosing specific behaviors.

Critical weaknesses
  1. Take Turn allows [0, 0.5] seconds for immediate entry, [0, 3] for delayed entry, and [-1, 0.5] for slight overlap (Appendix B.3), whereas Recover allows [-2, 5] seconds (B.9). Thus, the reported advantage for delayed entry and Recover cannot by itself distinguish adaptation to conversational conditions from ordinary response latency under much more permissive windows. Stop has no deadline: eventual termination of the current turn is immediate success, and follow-through becomes inapplicable if stopping occurs after the intervention ends (B.4). Interrupt can count as immediate success before the annotated intervention point, which is used only as a timing diagnostic (B.6). These definitions are disclosed, but they can inflate apparent immediate success or its gap to follow-through. The paper needs controls showing that its conclusions persist when success also reflects whether the required action was warranted and timely.

  2. The 97.9%/95.5% human-validation figures assess assigned action/condition labels of benchmark instances (Appendix A.4); they do not validate GPT-5.4's follow-through judgments, the audio-based backchannel judgment, or the timing-based classification of generated model speech. I did not find an independent audit of these output-level decisions. This is especially important for ambiguous requirements such as shortening or simplifying an ongoing response, and for heuristics treating all speech segments of at most three words as backchannels. The conditional follow-through scores can also be based on extremely few successes: Table 16 shows that GPT-Realtime-2.1's 100% affirmative-backchannel follow-through is based on one instance. Reporting those counts is helpful, but uncertainty and validation of the scoring process are still needed before interpreting such percentages as reliable capability measurements.

Questions for authors

See above

5 comments

We thank the reviewer for recognizing the value of our structured action taxonomy and the distinction between immediate execution and subsequent follow-through for diagnosing full-duplex interaction capabilities. We also appreciate the positive assessment of our broad action and scenario coverage, substantial real-world data, human validation, and evaluation across diverse models.

The revised and newly added content is marked in red in the PDF for convenience.


Weakness 1 Permissive timing windows for delayed Take Turn and Recover may confound adaptation with ordinary latency, while Stop and Interrupt criteria may count late or premature actions as successful. These choices could inflate Immediate success and its gap to Subsequent performance. Additional controls are needed to verify whether the conclusions hold under stricter timeliness and appropriateness criteria.

Response to Weakness 1: We thank the reviewer for this careful reading of our evaluation protocol, and understand the concern. Below we first clarify the design rationale, then report control analyses that address each concern.

Design rationale. Our action-specific timing windows are a deliberate design choice grounded in conversation analysis and interactional competence (Sec. 3.1 and Appendix A). The immediate Take Turn window ([0, 0.5] s) follows human turn-taking, where inter-turn gaps cluster around 200 ms (Stivers et al., 2009). The slight-overlap window ([−1.0, 0.5] s) admits the brief overlaps typical at turn transitions (Heldner & Edlund, 2010). The delayed window ([0, 3.0] s) reflects that humans also take turns with longer gaps depending on context, such as question type and response type (Strömbergsson et al., 2013).

For Recover, a longer window is inherent to the task: unlike Take Turn, the model must process an unexpected interruption, determine the required adjustment, and re-plan an ongoing contribution before resuming. The window [−2.0, 5.0] s accommodates both overlapping resumption and this re-planning time. In Table R3, this distinction is also observed in the model's different response latency between Recover and Take Turn (median 2.91 s for Recover vs. 1.35 s for Take Turn). For Stop, the binary criterion aims to record whether the model stops, while how quickly it yields is measured separately as Yield Latency (Table 4 and Table 18). Models differ mainly in the latter (0.96–5.20 s, Table 4), so late stops are not hidden but explicitly reported.

Table R3. Model-wise median response latencies for Take Turn and Recover and their differences (Recover − Take Turn) in seconds.

ModelTake TurnRecoverDifference
bayling-duplex3.264.20+0.94
flm-audio1.295.75+4.46
freeze-omni0.212.15+1.94
gemini-3.1-flash-live1.402.57+1.17
gpt-realtime-2.11.623.12+1.50
grok-voice1.823.17+1.35
minicpm-o-4.51.422.44+1.02
moshi0.391.78+1.39
qwen3.5-omni-flash-realtime1.142.69+1.55
raon-speechchat0.1711.80+11.63

For Interrupt, the authored reference point is used only as a timing diagnostic, because human conversation has no sharply defined moment at which an interruption becomes appropriate, and listeners can also recognize the need to intervene before the speaker reaches that point (Jefferson, 1984). Whether an interruption is warranted is therefore assessed by follow-through, which addresses the specific content that called for it.

References:

  • Stivers, T., et al. (2009). Universals and cultural variation in turn-taking in conversation. PNAS, 106(26), 10587–10592.
  • Heldner, M., et al. (2010). Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4), 555–568.
  • Strömbergsson, S., et al. (2013). Timing responses to questions in dialogue. Proc. Interspeech 2013.
  • Jefferson, G. (1984). Notes on some orderlinesses of overlap onset. In V. D'Urso & P. Leonardi (Eds.), Discourse Analysis and Natural Rhetoric (pp. 11–38). Padova: Cleup.

Continue response to Weakness 1:

Control analyses. To verify that our conclusions do not depend on these permissive choices, we re-scored all actions under stricter criteria.

(1) Take Turn and Recover: adaptation vs. ordinary latency. We agree that a higher score under a wider window does not by itself show adaptation. The paper does not claim it does. In fact, Section 5.1 argues the opposite: models do not adapt their timing to the conversational condition and instead follow a fixed turn-taking pattern. We further compute the latency over all trials and show that onsets in the delayed condition are nearly identical to those in the immediate condition (median 1.21 vs. 1.20 s).

Recover is different. If success there merely reflected slow responses, a model would be equally slow in Take Turn. Instead, all 10 models respond later in Recover than in Take Turn (median 2.91 vs. 1.35 s), with per-model latency differences (Recover minus Take Turn) ranging from +0.94 to +11.63 s. Models take extra time specifically when recovering from an intervention, and the wider Recover window captures this extra time rather than rewarding general slowness, which also reflects how humans recover in longer windows.

(2) Stop: late stops. We specifically reported yield latency to reflect the late stops, which ranges from 0.96 to 5.20 s across models (Table 4). To further address the reviewer's concern, we re-scored Stop by requiring termination before the user intervention ends, following the definition that the model should yield while the user holds the floor. Under this criterion, late stops after intervention ends count as failures, and follow-through is defined for all immediate success instances. Immediate success changes from 94.7–100% to 23.8–83.7%, and follow-through remains the same (since the original calculation only considers the cases that stop before user interruption ends), which is 70.7–100%. This is due to the short interruption period extracted from real-world conversations with 23% less than 2s. This setting is stricter than our original setting, therefore worse immediate success performance overall. We adopted this definition in the revision and updated the stop results accordingly.

(3) Interrupt: anchoring to the reference point. Table 4 shows that interruptions typically occur well before the reference point (timing difference −2.91 to −8.85 s across models). We therefore re-scored Interrupt with the window starting 1.0 s before the reference point, so that only interruptions made once the need for intervention has emerged count as success. Immediate success changes from 0.0–67.5% to 0.0–34.5%. We further compared follow-through between interruptions made before and after this point, pooled across models: 4.4% vs. 7.0%. Although overall performance decreases under the stricter window, the Spearman rank correlations for immediate and subsequent scores are 0.945 and 0.909, indicating that model rankings remain largely unchanged. And our conclusion still holds: almost none of the interruption behaviour subsequently carries out the required intervention.

We therefore retain our original definitions (except stop), which reflect the design rationale above, and have reported these analyses as robustness checks in newly revised Appendix F Table 21, together with the design rationale for each window in Appendix B.1. We also referenced them in Sec. 5.2, so that immediate success is interpreted alongside these stricter criteria.

Weakness 2: Subsequent evaluation lacks human validation of LLM-based subsequent judgement, audio-based backchannel judgement, and the ≤3-word heuristic for excluding short speech segments without independent validation, while small applicable sample sizes raise concerns about the reliability of reported success rates.

Response to Weakness 2:

  1. Validation results
    • Validation of GPT-5.4’s subsequent follow through judgement: We thank the reviewer for raising this concern. We sampled 454 instances from the seven actions (Stop is not included, which uses rule-based criteria) that require LLM judgment. YES and NO cases were balanced (217 vs. 237) so that agreement is not inflated by the more common label. Ten annotators were recruited, and each instance was labeled independently by two of them. Annotators saw the same input and instructions as GPT-5.4. For the 115 instances where the two disagreed, a third annotator gave a label, and the majority vote became the human reference.

      The validation results are shown in Table R4. GPT-5.4 agrees with the human reference on 77.1% of instances. This is on par with the 74.7% agreement between two human annotators. A judge cannot be expected to agree with humans more than humans agree with each other. The remaining disagreement reflects how subjective it is to judge pragmatic follow-through, rather than a weakness specific to GPT-5.4. The validation results and analysis are added to the main text section 5.4 and Appendix C.

      Table R4. Human–human agreement and GPT‑5.4 agreement with the human reference across 454 instances of seven actions.

      ActionHuman–Human AgreementGPT–Human Reference Agreement
      Wait75.0%82.1%
      Ignore72.0%82.9%
      Take Turn88.1%72.6%
      Backchannel72.7%77.3%
      Hold Floor66.7%75.0%
      Recover63.3%71.7%
      Interrupt82.5%75.0%
      Overall74.7%77.1%
    • Validation of audio-based backchannel judgement: We conducted additional human validation of the two LLM- or LALM-judge-based stages in Backchannel Immediate evaluation, each validated on about 10% of samples.

      (i) Backchannel verification (text-based): GPT-5.4 reads the candidate segment transcript and decides whether it is a backchannel, such as "mm-hm" or "yeah". It agrees with humans on 92.6% of 68 samples, close to the 95.6% human–human agreement.

      (ii) Function classification (audio-based): GPT-audio-1.5 model listens to each identified backchannel and decides whether it is affirmative (signaling agreement or "go on") or uncertain (signaling doubt). It agrees with humans on 71.1% of 38 samples, matching the 73.7% human–human agreement. The task is inherently subjective.

      In both steps, the judges agree with humans about as often as humans agree with each other, including audio-based and text-based judgements.

    • Clarification of heuristic for excluding short speech segments: For actions requiring substantive speech, such as Take Turn and Recover, we exclude segments of at most three words from qualifying onsets. This operational criterion distinguishes brief feedback from substantive turn-taking, following the general practice in prior work such as Candor and Full-duplex-bench V1.

Reviewer_iv8KOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DuplexAct presents a large-scale benchmark for evaluating interaction capabilities in full-duplex spoken language models. It defines 8 interaction actions (e.g., turn-taking, interruption, backchanneling) across 24 conditions. Unlike existing benchmarks, it evaluates both immediate execution and subsequent follow-through. The data is quality-checked by human annotators with high cross-validation agreement. The benchmark contains 4,691 instances (many more than existing benchmarks), including 54% real-world recordings (proportionally greater than most existing benchmarks), and evaluates 14 models. Results show that models often perform the correct immediate action but fail to respond appropriately afterward.

Critical strengths
  • The authors address important gaps in existing full-duplex benchmarks, especially the lack of systematic action coverage.
  • Well-structured taxonomy: Organizes conversational behaviors into 8 actions and 24 conditions.
  • Meaningful evaluation design: Separating immediate execution from subsequent follow-through provides a more complete and realistic assessment, revealing important shortcomings of existing full-duplex spoken language models.
  • Broad experiments: This paper evaluates 14 models across four conversational scenarios, providing important results for the community.
  • Real-world data: Includes 2,531 real conversational instances, with human validation of action and condition labels.
Critical weaknesses
  • Reliance on LLM judges: Performance on subsequent follow-through is evaluated using GPT-5.4, without a clear human validation of the judge's accuracy. Would the results change notably if the judge is changed?
  • Limited evaluation of long conversations: DuplexAct (the benchmark of this paper) primarily tests individual interaction actions rather than sustained, multi-turn conversational behavior.
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for recognizing the importance of addressing gaps in existing full-duplex benchmarks, particularly through our structured action taxonomy and the distinction between immediate execution and subsequent follow-through. We also appreciate reviwer’s positive assessment of our broad model evaluation, diverse conversational scenarios, and substantial real-world data coverage with human validation.

The revised and newly added content is marked in red in the PDF for convenience.


Weakenss 1: Lack of human validation and robustness analysis across different LLM judges.

Response to weakenss 1: We thank the reviewer for pointing out the concern of LLM judge reliability. We validated GPT-5.4's Subsequent judgments in two ways: against human annotators, and against two other LLM judges. The validation results and analysis are added to the main text section 5.4 and Appendix C.

  1. Agreement with human annotators.

We sampled 454 instances from the seven actions (Stop is not included, which uses rule-based criteria) that require LLM judgment. YES and NO cases were balanced (217 vs. 237) so that agreement is not inflated by the more common label. Ten annotators were recruited, and each instance was labeled independently by two of them. Annotators saw the same input and instructions as GPT-5.4. For the 115 instances where the two disagreed, a third annotator gave a label, and the majority vote became the human reference.

The validation results are shown in Table R1. GPT-5.4 agrees with the human reference on 77.1% of instances. This is on par with the 74.7% agreement between two human annotators. A judge cannot be expected to agree with humans more than humans agree with each other. The remaining disagreement reflects how subjective it is to judge pragmatic follow-through, rather than a weakness specific to GPT-5.4.

Table R1. Human–human agreement and GPT‑5.4 agreement with the human reference across 454 instances of seven actions.

ActionHuman–Human AgreementGPT–Human Reference Agreement
Wait75.0%82.1%
Ignore72.0%82.9%
Take Turn88.1%72.6%
Backchannel72.7%77.3%
Hold Floor66.7%75.0%
Recover63.3%71.7%
Interrupt82.5%75.0%
Overall74.7%77.1%
  1. Robustness to the choice of judge.

We re-judged all outputs with Claude Sonnet 5.5 and Gemini 3.8 Flash, using the same prompts. In Table R2, their verdicts match GPT-5.4's on 89.2% and 87.9% of all cases. Model rankings are largely preserved across judges, and all main findings of the paper hold under every judge.

Table R2. Cross-judge robustness of subsequent-follow-through judgments. S/G denotes Sonnet/Gemini.

ActionGPT–Sonnet agreementGPT–Gemini agreementRanking Spearman ρ (S / G)
Wait88.9%89.8%0.982 / 0.991
Ignore97.5%97.5%0.947 / 0.881
Take Turn86.6%83.6%0.978 / 0.960
Backchannel72.3%67.9%0.964 / 0.927
Hold Floor86.8%79.0%0.806 / 0.721
Recover87.0%88.0%0.976 / 0.988
Interrupt97.7%95.8%0.553 / 0.734
Overall89.2%87.9%—

Interrupt has lower rank correlations, but not because the judges disagree: verdict agreement is 97.7% and 95.8%. Most models have near-zero Subsequent success, so a single changed judgment can reorder them. All three judges agree on the key result: Subsequent success on Interrupt is uniformly low (GPT-5.4 5.0%, Sonnet 3.5%, Gemini 6.5%). Therefore, our finding in Sec. 5.2, that most interruptions which succeed immediately fail to carry out the intended intervention is unaffected.


Weakness 2: Focus on individual interaction actions rather than sustained multi-turn conversations.

Response to weakness 2: We thank the reviewer for highlighting this important direction. To our knowledge, existing full-duplex benchmarks still lack systematic coverage of interaction actions and success criteria. Our work takes a foundational step by establishing a structured action space and evaluating both immediate execution and subsequent follow-through across diverse interaction conditions. We believe such fine-grained, action-level benchmark provides an essential foundation for understanding more complex conversational dynamics. Investigating how these capabilities evolve and interact over sustained multi-turn conversations is an important next step, which we plan to explore in future work.

I appreciate the response and believe the paper makes an important contribution to related research communities. I have decided to maintime my positive score.

Dear Reviewer,

Thank you for your positive feedback and continued support of our work. We sincerely appreciate your thoughtful review and recognition of the contribution of DuplexAct to the research community.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Acknowledgement and Summary of Revisions

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the key contributions of DuplexAct, particularly (1) the structured, Interactional Competence-grounded taxonomy covering eight actions and 24 conditions, and (2) the distinction between Immediate execution and Subsequent follow-through, which reveals interaction failures overlooked by existing occurrence-based metrics. We also appreciate the positive assessments of our broad evaluation across 14 models, four conversational scenarios, and 4,691 instances, including 2,531 from real-world recordings.

In response to the reviewers' concerns, we have conducted substantial additional analyses and revised the manuscript accordingly:

  1. LLM-as-judge evaluation reliability (reviewers iv8k, kA7c, WDVa): We validated GPT-5.4's Subsequent judgments, achieving 77.1% agreement with the human reference, comparable to 74.7% human–human agreement. Re-evaluation using Claude Sonnet 5.5 and Gemini 3.8 Flash showed 89.2% and 87.9% agreement with GPT-5.4, with model rankings largely preserved. These are added in the main text section 5.4 and detailed in Appendix C. We additionally conducted human validation of the two judge-based stages in Backchannel Immediate evaluation: text-based verification achieved 92.6% agreement (vs. 95.6% human–human), while audio-based function classification using GPT-audio-1.5 achieved 71.1% (vs. 73.7% human–human).

     

  2. Evaluation criteria and control analyses (reviewer kA7c). We grounded the timing windows in conversation analysis (Appendix B.1, referenced in Section 5.1) and added latency analyses showing that the wider Recover window does not simply reward slow responses. We adopted a stricter criterion for Stop (Tables 3–4 and Appendix B.5), where the paper’s main claims remain unchanged. For Interrupt, we clarified the role of Interrupt (Appendix B.1, referenced in Section 5.1) and tested a stricter timing window as a robustness check. Model rankings are largely preserved (Spearman ρ > 0.9), and the main finding is unchanged (Appendix F, Table 21).

     

  3. Evaluation scope and methodology (reviewers iv8k, WDVa). Our fine-grained action-level evaluation provides a foundation for analyzing more complex conversational dynamics. We acknowledge that extending this evaluation to sustained multi-turn interactions is an important direction we plan to explore in future work.

     

    We clarified that DuplexAct's core contributions, including the action taxonomy, success criteria, and benchmark data, do not depend on the simulator's input modality and could apply directly to other audio-to-audio models. The current simulator uses streaming transcripts because it must identify semantically appropriate intervention points, which VAD or acoustic features alone cannot reliably detect. As audio-based simulators mature, they can replace this component without changing the benchmark design or its core contributions.

     

    We clarified that Safety and Help target users unaware of their risk or need, so the absence of distress prosody is intentional, and inferring intervention needs from content is the evaluated capability but not overfitting.

     

We thank the reviewers again for their time and valuable feedback. We welcome any further questions or concerns during the remaining discussion period and would be happy to address them promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.