- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
The paper points out that existing benchmarks for end-to-end full-duplex spoken language models (FD-SLMs) lack structured action-level definitions and overly focus on surface occurrences rather than pragmatic success. To address this, the authors propose DUPLEXACT, a benchmark grounded in human Interactional Competence, featuring 8 actions and 24 conditions. This benchmark strictly divides action evaluation into two independent stages: "immediate execution" and "subsequent follow-through". The dataset contains 4,691 instances, combining real-world corpora and controlled synthesized data. Evaluations of 14 frontier models reveal a systemic deficiency in understanding the deeper semantic consequences of interactive actions.
Critical strengths
- Evaluation Dimensions: The paper extends evaluation beyond metrics such as response latency and takeover rates by introducing the “subsequent follow-through” metric. This metric captures cases where models interrupt at an appropriate time but do not provide an appropriate response afterward.
- Taxonomy: The classification of 8 actions and 24 conditions, based on Interactional Competence, provides a comprehensive framework for studying full-duplex dialogue control.
Critical weaknesses
- Validation of Automated Metrics: The success of “subsequent follow-through” is evaluated using an LLM judge, including whether the model simplified the language, revised obsolete content, or shifted the topic. The paper reports human annotator agreement (97.9% and 95.5%) during dataset construction but does not report agreement between the LLM judge’s scores and human judgments. The reliability of the automated evaluation therefore remains unclear.
- Evaluation Pipeline Requirements: Simulator Replay (covering Stop, Hold Floor, Recover, and Interrupt) requires streaming text transcripts aligned with audio progress. This requirement prevented the evaluation of 4 of the 14 tested open-source models. It may also limit the benchmark’s applicability to audio-to-audio models that do not produce text transcripts.
- Acoustic Properties of Synthesized Data: For L2 conditions involving high cognitive load, such as “Safety,” the paper uses an LLM-to-TTS synthesis pipeline. Emergency interruptions can involve changes in speech rate and pitch, as well as overlapping speech. Whether the synthesized data reproduce these acoustic features is unclear, leaving the ecological validity of these conditions unverified.
Questions for authors
- As the core innovation of this benchmark, the "subsequent follow-through" evaluation relies heavily on an LLM judge's assessment of pragmatic consistency. Did the authors conduct any Human-LLM Agreement validation (e.g., Cohen's Kappa) on a subset of the test set? Without this validation, how can the community trust that the benchmark scores accurately reflect true interactive performance?
- Given that Simulator Replay heavily relies on streaming text transcripts, how can the authors prove that DUPLEXACT will be able to adapt to future pure audio-to-audio E2E models that discard intermediate text representations? Have you considered trigger alternatives based on low-latency VAD or acoustic features?
- Regarding the TTS-synthesized interruption data for "Safety" and "Help" scenarios, how do the authors ensure that the generated speech contains the unique paralinguistic and prosodic features typical of humans facing danger or needing urgent assistance? Does this lead to models merely overfitting to specific text semantics rather than learning authentic acoustic interaction dynamics?
4 comments
Weakness 3 and Question 3: TTS-generated Safety and Help scenarios may lack realistic prosodic and paralinguistic cues, raising concerns about ecological validity and overreliance on text semantics.
Response to Weakness 3 and Question 3: Our Safety and Help conditions target a different case: users who are unaware of a risk or of their need for help. In these cases, there is no urgency or distress in the user's voice to detect. The model must infer the need to intervene from what the user is saying. For instance:
- Safety: A user calmly mentions, "I found a small mercury thermometer broken in the nursery dresser, and I'm vacuuming up the shiny beads from the carpet..." Unaware of the danger, the user continues speaking, and the model should proactively interrupt to warn against vacuuming mercury.
- Help: A user may calmly discuss several travel options, going back and forth between them without showing the hesitant tones to ask for help. The model should step in with a suggestion.
Distress prosody is therefore absent by design, not a limitation of TTS. Relying on content in these cases is not overfitting to text semantics. Instead, recognizing a risk or a need for help that is not acoustically signaled is precisely the capability these conditions evaluate (Table 7).
Response to Authors
Dear authors,
Thank you for the substantial additional work during the rebuttal period and your clarifications on the streaming-transcript requirement and the design of the Safety and Help scenarios. Since full-duplex dialogue involves human–machine interaction, my main concern lies on whether the LLM score can truly reflect human recognition of this benchmark score. The human alignment experiment supplemented by the authors have addressed my concerns. I will update my review and score accordingly.
Dear Reviewer,
Thank you for your positive feedback and for acknowledging our additional efforts. We sincerely appreciate your thoughtful comments and your willingness to reconsider and increase the score.
We thank the reviewer for recognizing the value of our Subsequent follow-through evaluation in revealing limitations overlooked by existing metrics, particularly models' failure to fulfill the intended purpose of their actions. We also appreciate reviewer’s positive assessment of our structured, Interactional Competence-grounded taxonomy, which provides a systematic framework for evaluating full-duplex interaction capabilities.
The revised and newly added content is marked in red in the PDF for convenience.
Weakness 1 and Question 1: Subsequent follow-through relies on LLM judgments without human–LLM agreement validation, raising concerns about the reliability of benchmark scores.
Response to Weakness 1 and Question 1: We thank the reviewer for raising this concern. We sampled 454 instances from the seven actions (Stop is not included, which uses rule-based criteria) that require LLM judgment. YES and NO cases were balanced (217 vs. 237) so that agreement is not inflated by the more common label. Ten annotators were recruited, and each instance was labeled independently by two of them. Annotators saw the same input and instructions as GPT-5.4. For the 115 instances where the two disagreed, a third annotator gave a label, and the majority vote became the human reference.
The validation results are shown in Table R7. GPT-5.4 agrees with the human reference on 77.1% of instances. This is on par with the 74.7% agreement between two human annotators. A judge cannot be expected to agree with humans more than humans agree with each other. The remaining disagreement reflects how subjective it is to judge pragmatic follow-through, rather than a weakness specific to GPT-5.4.
Table R7. Human–human agreement and GPT‑5.4 agreement with the human reference across 454 instances of seven actions.
Weakness 2 and Question 2: Simulator Replay requires intermediate streaming text representation, limiting model coverage and raising concerns about compatibility with future pure audio-to-audio models. Could low-latency VAD or acoustic features serve as alternative triggers?
Response to Weakness 2 and Question 2: The key contributions of our work is the action taxonomy, success criteria, and benchmark data coverage, which are applicable to pure audio-to-audio models. The reliance on intermediate streaming transcripts is a deliberate choice in the current simulator implementation.
Simulator Replay must inject recorded user speech at contextually appropriate points in the model's ongoing response (e.g., Recover and Hold Floor). Since different models generate different content, these points cannot be fixed in advance and require semantic understanding of the ongoing response. Following τ-Voice, we adopt an LLM-driven simulator using intermediate streaming transcripts to make low-latency interaction decisions. VAD or acoustic features alone cannot reliably identify these semantically appropriate intervention points.
Applying external streaming ASR to the model's output audio is a possible alternative, but its additional latency and transcription accuracy would further shift the injection points and compromise comparability across models. Therefore, the resilience on the streaming text transcripts is currently the most practical choice.
We agree that full-duplex models may move toward pure audio-to-audio processing, and the simulator can evolve in the same direction by taking the model's output audio directly as input with better semantic understanding, low-latency decision-making, and more controllable speech output content. As they mature, the text-based module can be replaced without affecting the benchmark design, since the contributions are independent of the simulator's input modality.