- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark called Duplex-MPE which evaluates full-duplex models on their selective participation in conversations involving multiple user speakers. Currently, existing benchmarks focus on dyadic interactions between a user and assistant, however this benchmark evaluates whether an assistant can identify requests addressed to it (explicit and implicit), remain silent during non-addressed speech and stop speaking when another user resolves a request.
The main contribution is the 2000 synthetic multi party conversation scenarios, each with explicit and implicit addressing resulting in 4000 audio streams. The authors further contribute four evaluation metrics which are (i) fresh response initiation within 5 seconds, (ii) answer correctness when responding, (iii) silence during non-addressed speech, and (iv) stopping within 2 seconds when another participant resolves a request.
Finally, the authors evaluate 5 open-weight speech models, showing that MiniCPM-o 4.5 is ahead while other systems frequently exhibit unwanted speech or incorrect responses. They also find that explicit versus implicit addressing has a substantial effect on the text-based reference but no statistically significant effect on the speech systems' response rates.
Critical strengths
-
The proposed metrics are decomposed into 4 criteria, and this decomposition reveals qualitatively different failure modes that would be hidden by conventional response-rate metrics. For example, Table 3 shows that Freeze-Omni achieves nearly perfect response presence while almost completely failing to preserve silence, whereas MiniCPM-o exhibits substantially more selective responses. This is complemented with additional human validation, uncertainty estimates, and sensitivity analyses.
-
The evaluation setting is relatively underexplored, hence the paper addresses a meaningful gap in existing spoken dialogue evaluation.
Critical weaknesses
-
The benchmark relies on only four fixed TTS male voices, with clean, sequential human turns and no overlapping human speech. This setup substantially limits its representation of realistic multi-party conversations, which involve diverse voices, spontaneous interruptions, simultaneous speech, and varying acoustic conditions. Moreover, the fixed speaker-to-voice mapping may artificially simplify speaker identification. These limitations are particularly concerning given that the benchmark aims to evaluate multi-party conversational understanding, and raise questions about whether the reported results generalize to more natural interactions.
-
The evaluation metrics use different, model-dependent denominators, making cross-model comparisons difficult. In particular, answering-window yield only evaluates cases where the model starts responding within a specific interval and is still speaking when the human resolves the request. For example, Voila's 84.8% yield rate is computed over only 620 eligible events, compared with 1,633 for MiniCPM-o.
-
Gemini 3.1 Pro responds to 94.7% of explicitly addressed requests but only 30.4% of implicit requests, whereas MiniCPM-o responds to 95.5% and 94.9%, respectively. The authors highlight this contrast, but it is unclear whether MiniCPM-o successfully understands implicit addressing or simply tends to answer questions regardless of whom they are directed to. For example, a model that answers nearly every question would also achieve similar response rates in both conditions, without demonstrating any addressee understanding. While the silence-preservation results provide some counterevidence, the experiment does not conclusively distinguish these explanations. Furthermore, the paired scenarios contain additional wording and structural differences (Appendix A2.4), making the addressing comparison less controlled.
Questions for authors
-
Appendix A3.1 specifies a 31.9-second spoken preamble instructing models when to respond or remain silent. How sensitive are the results to this instruction? Have the authors tested alternative phrasings or shorter instructions?
-
Table 3 shows extremely low conditional answer accuracy for four of the five models (below 2%), compared with approximately 47% for MiniCPM-o. Can the authors explain these unusually large differences? Are they primarily caused by incompatibilities between the models' expected interaction protocols and the evaluation setup?
-
The protocol uses fixed response deadlines of 5 seconds for T requests and 3 seconds before human resolution in N4 events. How were these thresholds selected and how sensitive are the results and model rankings to these choices? Could differences in response latency or speaking duration, rather than conversational understanding, account for some of the observed performance gaps?
11 comments
| Model | Acoustic & Turn-Taking Condition ( scenarios, 1,297 turns) | Fresh-Onset ↑ | RP | Cond. AA ↑ | Uncond. AA (/100) ↑ | Silence Pres. (SP) ↑ | Answering Yield (AY) ↑ |
|---|---|---|---|---|---|---|---|
| MiniCPM-o 4.5 | 1. Original Baseline (4 Adult Male Voices, Clean) | 97.0% | 97.0% | 48.5% | 47.0% | 92.6% | 61.0% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 92.0% | 93.0% | 43.0% | 40.0% | 93.6% | 58.2% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 82.0% | 82.0% | 43.9% | 36.0% | 70.3% | 62.5% | |
| Realtime-Venus-Audio | 1. Original Baseline (4 Adult Male Voices, Clean) | 86.0% | 94.0% | 21.3% | 20.0% | 60.2% | 41.5% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 89.0% | 95.0% | 21.1% | 20.0% | 68.3% | 35.4% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 90.0% | 93.0% | 15.1% | 14.0% | 62.0% | 32.7% | |
| Voila | 1. Original Baseline (4 Adult Male Voices, Clean) | 77.0% | 77.0% | 1.3% | 1.0% | 83.5% | 90.9% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 79.0% | 81.0% | 0.0% | 0.0% | 85.0% | 94.2% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 72.0% | 72.0% | 0.0% | 0.0% | 72.2% | 92.2% | |
| Moshika-RL-Seamless | 1. Original Baseline (4 Adult Male Voices, Clean) | 89.0% | 100.0% | 4.0% | 4.0% | 47.8% | 7.7% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 87.0% | 99.0% | 2.0% | 2.0% | 34.1% | 0.0% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 90.0% | 100.0% | 3.0% | 3.0% | 40.9% | 0.0% | |
| PersonaPlex | 1. Original Baseline (4 Adult Male Voices, Clean) | 75.0% | 96.0% | 2.1% | 2.0% | 4.2% | 0.0% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 84.0% | 99.0% | 2.0% | 2.0% | 4.4% | 0.0% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 86.0% | 99.0% | 1.0% | 1.0% | 3.5% | 0.0% | |
| Moshi | 1. Original Baseline (4 Adult Male Voices, Clean) | 71.0% | 89.0% | 1.1% | 1.0% | 27.5% | 26.7% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 60.0% | 71.0% | 0.0% | 0.0% | 26.2% | 11.1% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 68.0% | 80.0% | 1.3% | 1.0% | 31.1% | 15.4% | |
| FLM-Audio | 1. Original Baseline (4 Adult Male Voices, Clean) | 53.0% | 93.0% | 0.0% | 0.0% | 39.5% | 1.9% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 54.0% | 98.0% | 0.0% | 0.0% | 47.9% | 0.0% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 43.0% | 94.0% | 0.0% | 0.0% | 29.9% | 0.0% | |
| Freeze-Omni | 1. Original Baseline (4 Adult Male Voices, Clean) | 1.0% | 2.0% | 0.0% | 0.0% | 84.7% | 0.0% |
| 2. Diverse Voices (2 Female + Child + Elderly, Clean) | 0.0% | 6.0% | 0.0% | 0.0% | 77.7% | 0.0% | |
| 3. Diverse Voices + Human-Human Overlap (Latching + Backchannels) | 1.0% | 9.0% | 0.0% | 0.0% | 77.6% | 0.0% |
Key findings across all 8 models: (i) Replacing adult male voices with female, child, and elderly voices leaves model rankings and capability profiles intact (e.g., MiniCPM-o 4.5 achieves 93.6% SP vs. 92.6%, 58.2% AY vs. 61.0%, and 43.0% Cond. AA vs. 48.5%; Realtime-Venus-Audio achieves 21.1% Cond. AA / 20.0% Uncond. AA vs. 21.3% / 20.0% baseline and 68.3% SP; Voila achieves 85.0% SP and 94.2% AY; while all other five models remain % AA); (ii) Adding human-human speech overlap further degrades turn-boundary detection and silence preservation: MiniCPM-o 4.5 maintains 43.9% Cond. AA when it responds, but its Fresh-Onset drops by 10.0 pp (92.0% 82.0%) and its SP drops by 23.3 pp (93.6% 70.3%); similarly, Voila drops 7.0 pp in Fresh-Onset (79.0% 72.0%) and 12.8 pp in SP (85.0% 72.2%), FLM-Audio drops 18.0 pp in SP (47.9% 29.9%), and Realtime-Venus-Audio drops 6.0 pp in Cond. AA (21.1% 15.1%) and 6.3 pp in SP (68.3% 62.0%). We have added these complete 8-model experiments to Appendix Table 26 of the revised manuscript.
W2
"The evaluation metrics use different, model-dependent denominators, making cross-model comparisons difficult. In particular, answering-window yield only evaluates cases where the model starts responding within a specific interval and is still speaking when the human resolves the request. For example, Voila's 84.8% yield rate is computed over only 620 eligible events, compared with 1,633 for MiniCPM-o."
We completely agree that conditional rates (AA and AY) should be accompanied by fixed-denominator (unconditional) rates so readers can compare models on identical event sets. In the revised manuscript, we provide both conditional and fixed-denominator rates across all 8 models: specifically, Unconditional AA keeps the same numerator as AA while fixing the denominator to all 2,000 T requests (measuring the joint ability to respond and answer correctly), and Unconditional AY () keeps the exact same numerator as AY while fixing the denominator to all 1,911 N4 events (measuring the joint ability to both initiate a timely response after and stop in time when begins):
| Model | Cond. | RP (/2000) | Cond. AA (Num/Den) | Uncond. AA (/2000 T) | Cond. AY (Num/Den) | Uncond. Yield (/1911 N4) | Timely Start & Silent After (/1911 N4) |
|---|---|---|---|---|---|---|---|
| MiniCPM-o 4.5 | Explicit | 95.50% | 45.29% (865/1910) | 43.25% (865/2000) | 56.29% (935/1661) | 48.93% (935/1911) | 53.22% (1017/1911) |
| Implicit | 94.30% | 46.45% (876/1886) | 43.80% (876/2000) | 55.86% (925/1656) | 48.40% (925/1911) | 52.80% (1009/1911) | |
| Realtime-Venus-Audio | Explicit | 91.10% | 23.44% (427/1822) | 21.35% (427/2000) | 36.30% (334/920) | 17.48% (334/1911) | 19.41% (371/1911) |
| Implicit | 91.90% | 23.18% (426/1838) | 21.30% (426/2000) | 36.79% (351/954) | 18.37% (351/1911) | 19.99% (382/1911) | |
| Voila | Explicit | 80.50% | 1.24% (20/1610) | 1.00% (20/2000) | 90.77% (934/1029) | 48.87% (934/1911) | 54.63% (1044/1911) |
| Implicit | 81.70% | 1.65% (27/1634) | 1.35% (27/2000) | 89.73% (926/1032) | 48.46% (926/1911) | 54.37% (1039/1911) | |
| Moshika-RL-Seamless | Explicit | 99.30% | 1.96% (39/1986) | 1.95% (39/2000) | 5.56% (12/216) | 0.63% (12/1911) | 1.41% (27/1911) |
| Implicit | 99.40% | 2.21% (44/1988) | 2.20% (44/2000) | 4.55% (11/242) | 0.58% (11/1911) | 0.78% (15/1911) | |
| PersonaPlex | Explicit | 98.95% | 2.17% (43/1979) | 2.15% (43/2000) | 1.32% (1/76) | 0.05% (1/1911) | 0.31% (6/1911) |
| Implicit | 99.05% | 1.72% (34/1981) | 1.70% (34/2000) | 3.03% (2/66) | 0.10% (2/1911) | 0.31% (6/1911) | |
| Moshi | Explicit | 81.50% | 1.41% (23/1630) | 1.15% (23/2000) | 19.00% (42/221) | 2.20% (42/1911) | 3.56% (68/1911) |
| Implicit | 82.20% | 1.58% (26/1644) | 1.30% (26/2000) | 21.16% (51/241) | 2.67% (51/1911) | 4.66% (89/1911) | |
| FLM-Audio | Explicit | 95.05% | 0.11% (2/1901) | 0.10% (2/2000) | 0.82% (8/977) | 0.42% (8/1911) | 0.58% (11/1911) |
| Implicit | 95.45% | 0.05% (1/1909) | 0.05% (1/2000) | 0.59% (6/1018) | 0.31% (6/1911) | 0.47% (9/1911) | |
| Freeze-Omni | Explicit | 4.25% | 0.00% (0/85) | 0.00% (0/2000) | 6.45% (6/93) | 0.31% (6/1911) | 0.31% (6/1911) |
| Implicit | 4.55% | 0.00% (0/91) | 0.00% (0/2000) | 2.30% (2/87) | 0.10% (2/1911) | 0.16% (3/1911) |
Key takeaway: After fixing Voila's streaming codec state, its eligible N4 denominator increases from 620 to 1,029 (Explicit) / 1,032 (Implicit), and its raw count of successful yields (934 / 926) is virtually identical to MiniCPM-o's (935 / 925, or 48.9% vs. 48.9% of all 1,911 N4 events). Reporting both conditional and fixed-denominator views makes cross-model comparison completely transparent.
We thank Reviewer
HBEHfor the thorough, expert review and insightful questions on instruction sensitivity, model-dependent denominators, addressing controls, and timing thresholds. Below we address every weakness and question with concrete quantitative analyses.W1
We agree that real-world meetings include diverse voice timbres, acoustic noise, and human-human overlap, and we explicitly note this scope in our Ethics/Limitations statement. We designed Duplex-MPE's acoustic layer with clean, distinct voices for two diagnostic reasons:
N4_Rresolution and everyN1–N3intrusion involves real simultaneous human-and-assistant speech.Qwen3-TTS-CustomVoice:vivian, Speaker A), a warm adult female voice (serena, Speaker B), a 10-year-old child voice (ono_annawith high-pitched child instruction, Speaker C), and a 70-year-old elderly voice (uncle_fuwith slow aged instruction, Speaker D).