acceptodds
Under review as a conference paper at ICLR 2027

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

24 views

Abstract

Real-time full-duplex speech models can listen while speaking, enabling natural interaction without rigid turn boundaries. Existing benchmarks evaluate turn-taking, interruption handling and multi-round dialogue, but largely centre on a designated user rather than an assistant participating in a shared conversation among several people. We introduce Duplex-MPE to evaluate when such an assistant should answer, remain silent or stop speaking. The benchmark contains 2,000 scenarios with three or four human speakers and one assistant, each paired across explicit and implicit addressing of the same request. Models receive continuous conversation audio without transcripts or supplied turn boundaries. Four scores measure fresh response initiation, answer accuracy, silence preservation and stopping when a human resolves a request. We evaluate five open-weight speech systems: MiniCPM-o 4.5, Moshi, FLM-Audio, Voila and Freeze-Omni. MiniCPM-o 4.5 leads on three scored capabilities, while frequent speech from other systems can coexist with inaccurate answers or failures to remain silent. A transcript-based Gemini 3.1 Pro reference responds 64.3 percentage points more often to explicit than implicit requests; paired tests detect no significant response-rate difference for the speech systems. Audio examples are provided in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
full-duplex dialogue, multi-party conversation, speech language models, benchmark, turn-taking, addressee recognition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HBEHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark called Duplex-MPE which evaluates full-duplex models on their selective participation in conversations involving multiple user speakers. Currently, existing benchmarks focus on dyadic interactions between a user and assistant, however this benchmark evaluates whether an assistant can identify requests addressed to it (explicit and implicit), remain silent during non-addressed speech and stop speaking when another user resolves a request.

The main contribution is the 2000 synthetic multi party conversation scenarios, each with explicit and implicit addressing resulting in 4000 audio streams. The authors further contribute four evaluation metrics which are (i) fresh response initiation within 5 seconds, (ii) answer correctness when responding, (iii) silence during non-addressed speech, and (iv) stopping within 2 seconds when another participant resolves a request.

Finally, the authors evaluate 5 open-weight speech models, showing that MiniCPM-o 4.5 is ahead while other systems frequently exhibit unwanted speech or incorrect responses. They also find that explicit versus implicit addressing has a substantial effect on the text-based reference but no statistically significant effect on the speech systems' response rates.

Critical strengths
  1. The proposed metrics are decomposed into 4 criteria, and this decomposition reveals qualitatively different failure modes that would be hidden by conventional response-rate metrics. For example, Table 3 shows that Freeze-Omni achieves nearly perfect response presence while almost completely failing to preserve silence, whereas MiniCPM-o exhibits substantially more selective responses. This is complemented with additional human validation, uncertainty estimates, and sensitivity analyses.

  2. The evaluation setting is relatively underexplored, hence the paper addresses a meaningful gap in existing spoken dialogue evaluation.

Critical weaknesses
  1. The benchmark relies on only four fixed TTS male voices, with clean, sequential human turns and no overlapping human speech. This setup substantially limits its representation of realistic multi-party conversations, which involve diverse voices, spontaneous interruptions, simultaneous speech, and varying acoustic conditions. Moreover, the fixed speaker-to-voice mapping may artificially simplify speaker identification. These limitations are particularly concerning given that the benchmark aims to evaluate multi-party conversational understanding, and raise questions about whether the reported results generalize to more natural interactions.

  2. The evaluation metrics use different, model-dependent denominators, making cross-model comparisons difficult. In particular, answering-window yield only evaluates cases where the model starts responding within a specific interval and is still speaking when the human resolves the request. For example, Voila's 84.8% yield rate is computed over only 620 eligible events, compared with 1,633 for MiniCPM-o.

  3. Gemini 3.1 Pro responds to 94.7% of explicitly addressed requests but only 30.4% of implicit requests, whereas MiniCPM-o responds to 95.5% and 94.9%, respectively. The authors highlight this contrast, but it is unclear whether MiniCPM-o successfully understands implicit addressing or simply tends to answer questions regardless of whom they are directed to. For example, a model that answers nearly every question would also achieve similar response rates in both conditions, without demonstrating any addressee understanding. While the silence-preservation results provide some counterevidence, the experiment does not conclusively distinguish these explanations. Furthermore, the paired scenarios contain additional wording and structural differences (Appendix A2.4), making the addressing comparison less controlled.

Questions for authors
  1. Appendix A3.1 specifies a 31.9-second spoken preamble instructing models when to respond or remain silent. How sensitive are the results to this instruction? Have the authors tested alternative phrasings or shorter instructions?

  2. Table 3 shows extremely low conditional answer accuracy for four of the five models (below 2%), compared with approximately 47% for MiniCPM-o. Can the authors explain these unusually large differences? Are they primarily caused by incompatibilities between the models' expected interaction protocols and the evaluation setup?

  3. The protocol uses fixed response deadlines of 5 seconds for T requests and 3 seconds before human resolution in N4 events. How were these thresholds selected and how sensitive are the results and model rankings to these choices? Could differences in response latency or speaking duration, rather than conversational understanding, account for some of the observed performance gaps?

11 comments

We thank Reviewer HBEH for the thorough, expert review and insightful questions on instruction sensitivity, model-dependent denominators, addressing controls, and timing thresholds. Below we address every weakness and question with concrete quantitative analyses.

W1

"The benchmark relies on only four fixed TTS male voices, with clean, sequential human turns and no overlapping human speech... Moreover, the fixed speaker-to-voice mapping may artificially simplify speaker identification."

We agree that real-world meetings include diverse voice timbres, acoustic noise, and human-human overlap, and we explicitly note this scope in our Ethics/Limitations statement. We designed Duplex-MPE's acoustic layer with clean, distinct voices for two diagnostic reasons:

  1. Preventing speaker-confusion from confounding multi-party pragmatic evaluation: As you noted, fixed distinct voices make speaker differentiation acoustically easy. Crucially, this strengthens our empirical finding (a fortiori): even when speaker differentiation and acoustic clarity are maximally simplified, 6 of the 8 evaluated full-duplex models achieve <2.5%<2.5\%% answer accuracy and 5 of 8 fail to preserve silence (4.0%–60.9% SP). This proves that current full-duplex models fail not because of front-end speaker diarization or noise, but because they lack multi-party addressee control and cross-turn context tracking.
  2. Dynamic human–assistant overlap is still tested: Although human turns do not overlap each other, human audio streams continuously while the assistant is speaking—meaning every N4_R resolution and every N1–N3 intrusion involves real simultaneous human-and-assistant speech.
  3. New supplementary experiments with Female, Child, and Elderly Voices + Human-Human Overlapping Speech (Appendix Table 26): To directly address your concern with empirical evidence, we re-synthesized the stratified N=100N=100 scenario subset (1,297 turns) under two new conditions using Qwen3-TTS-CustomVoice:
    • Condition 2A (Diverse Demographic Voices — Female, Child, Elderly, Clean): Replacing the four male voices with a young female voice (vivian, Speaker A), a warm adult female voice (serena, Speaker B), a 10-year-old child voice (ono_anna with high-pitched child instruction, Speaker C), and a 70-year-old elderly voice (uncle_fu with slow aged instruction, Speaker D).
    • Condition 2B (Diverse Voices + Human-Human Overlapping Speech): Building on Condition 2A by injecting 739 turn-transition overlaps (350 ms early entry / latching) between consecutive non-T turns by different human speakers and 829 mid-utterance backchannel overlaps ("Mm-hmm", "Yeah, right" at 0.55×0.55\times gain) during non-T turns >3.5 s>3.5\text{ s}.
ModelAcoustic & Turn-Taking Condition (N=100N=100 scenarios, 1,297 turns)Fresh-Onset ↑RPCond. AA ↑Uncond. AA (/100) ↑Silence Pres. (SP) ↑Answering Yield (AY) ↑
MiniCPM-o 4.51. Original Baseline (4 Adult Male Voices, Clean)97.0%97.0%48.5%47.0%92.6%61.0%
2. Diverse Voices (2 Female + Child + Elderly, Clean)92.0%93.0%43.0%40.0%93.6%58.2%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)82.0%82.0%43.9%36.0%70.3%62.5%
Realtime-Venus-Audio1. Original Baseline (4 Adult Male Voices, Clean)86.0%94.0%21.3%20.0%60.2%41.5%
2. Diverse Voices (2 Female + Child + Elderly, Clean)89.0%95.0%21.1%20.0%68.3%35.4%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)90.0%93.0%15.1%14.0%62.0%32.7%
Voila1. Original Baseline (4 Adult Male Voices, Clean)77.0%77.0%1.3%1.0%83.5%90.9%
2. Diverse Voices (2 Female + Child + Elderly, Clean)79.0%81.0%0.0%0.0%85.0%94.2%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)72.0%72.0%0.0%0.0%72.2%92.2%
Moshika-RL-Seamless1. Original Baseline (4 Adult Male Voices, Clean)89.0%100.0%4.0%4.0%47.8%7.7%
2. Diverse Voices (2 Female + Child + Elderly, Clean)87.0%99.0%2.0%2.0%34.1%0.0%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)90.0%100.0%3.0%3.0%40.9%0.0%
PersonaPlex1. Original Baseline (4 Adult Male Voices, Clean)75.0%96.0%2.1%2.0%4.2%0.0%
2. Diverse Voices (2 Female + Child + Elderly, Clean)84.0%99.0%2.0%2.0%4.4%0.0%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)86.0%99.0%1.0%1.0%3.5%0.0%
Moshi1. Original Baseline (4 Adult Male Voices, Clean)71.0%89.0%1.1%1.0%27.5%26.7%
2. Diverse Voices (2 Female + Child + Elderly, Clean)60.0%71.0%0.0%0.0%26.2%11.1%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)68.0%80.0%1.3%1.0%31.1%15.4%
FLM-Audio1. Original Baseline (4 Adult Male Voices, Clean)53.0%93.0%0.0%0.0%39.5%1.9%
2. Diverse Voices (2 Female + Child + Elderly, Clean)54.0%98.0%0.0%0.0%47.9%0.0%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)43.0%94.0%0.0%0.0%29.9%0.0%
Freeze-Omni1. Original Baseline (4 Adult Male Voices, Clean)1.0%2.0%0.0%0.0%84.7%0.0%
2. Diverse Voices (2 Female + Child + Elderly, Clean)0.0%6.0%0.0%0.0%77.7%0.0%
3. Diverse Voices + Human-Human Overlap (Latching + Backchannels)1.0%9.0%0.0%0.0%77.6%0.0%

Key findings across all 8 models: (i) Replacing adult male voices with female, child, and elderly voices leaves model rankings and capability profiles intact (e.g., MiniCPM-o 4.5 achieves 93.6% SP vs. 92.6%, 58.2% AY vs. 61.0%, and 43.0% Cond. AA vs. 48.5%; Realtime-Venus-Audio achieves 21.1% Cond. AA / 20.0% Uncond. AA vs. 21.3% / 20.0% baseline and 68.3% SP; Voila achieves 85.0% SP and 94.2% AY; while all other five models remain ≤2.0\le 2.0% AA); (ii) Adding human-human speech overlap further degrades turn-boundary detection and silence preservation: MiniCPM-o 4.5 maintains 43.9% Cond. AA when it responds, but its Fresh-Onset drops by 10.0 pp (92.0% →\rightarrow 82.0%) and its SP drops by 23.3 pp (93.6% →\rightarrow 70.3%); similarly, Voila drops 7.0 pp in Fresh-Onset (79.0% →\rightarrow 72.0%) and 12.8 pp in SP (85.0% →\rightarrow 72.2%), FLM-Audio drops 18.0 pp in SP (47.9% →\rightarrow 29.9%), and Realtime-Venus-Audio drops 6.0 pp in Cond. AA (21.1% →\rightarrow 15.1%) and 6.3 pp in SP (68.3% →\rightarrow 62.0%). We have added these complete 8-model experiments to Appendix Table 26 of the revised manuscript.

W2

"The evaluation metrics use different, model-dependent denominators, making cross-model comparisons difficult. In particular, answering-window yield only evaluates cases where the model starts responding within a specific interval and is still speaking when the human resolves the request. For example, Voila's 84.8% yield rate is computed over only 620 eligible events, compared with 1,633 for MiniCPM-o."

We completely agree that conditional rates (AA and AY) should be accompanied by fixed-denominator (unconditional) rates so readers can compare models on identical event sets. In the revised manuscript, we provide both conditional and fixed-denominator rates across all 8 models: specifically, Unconditional AA keeps the same numerator as AA while fixing the denominator to all 2,000 T requests (measuring the joint ability to respond and answer correctly), and Unconditional AY (Yielded/1,911\text{Yielded} / 1{,}911) keeps the exact same numerator as AY while fixing the denominator to all 1,911 N4 events (measuring the joint ability to both initiate a timely response after N4Q\text{N4}_Q and stop in time when N4R\text{N4}_R begins):

ModelCond.RP (/2000)Cond. AA (Num/Den)Uncond. AA (/2000 T)Cond. AY (Num/Den)Uncond. Yield (/1911 N4)Timely Start & Silent After N4R\text{N4}_R (/1911 N4)
MiniCPM-o 4.5Explicit95.50%45.29% (865/1910)43.25% (865/2000)56.29% (935/1661)48.93% (935/1911)53.22% (1017/1911)
Implicit94.30%46.45% (876/1886)43.80% (876/2000)55.86% (925/1656)48.40% (925/1911)52.80% (1009/1911)
Realtime-Venus-AudioExplicit91.10%23.44% (427/1822)21.35% (427/2000)36.30% (334/920)17.48% (334/1911)19.41% (371/1911)
Implicit91.90%23.18% (426/1838)21.30% (426/2000)36.79% (351/954)18.37% (351/1911)19.99% (382/1911)
VoilaExplicit80.50%1.24% (20/1610)1.00% (20/2000)90.77% (934/1029)48.87% (934/1911)54.63% (1044/1911)
Implicit81.70%1.65% (27/1634)1.35% (27/2000)89.73% (926/1032)48.46% (926/1911)54.37% (1039/1911)
Moshika-RL-SeamlessExplicit99.30%1.96% (39/1986)1.95% (39/2000)5.56% (12/216)0.63% (12/1911)1.41% (27/1911)
Implicit99.40%2.21% (44/1988)2.20% (44/2000)4.55% (11/242)0.58% (11/1911)0.78% (15/1911)
PersonaPlexExplicit98.95%2.17% (43/1979)2.15% (43/2000)1.32% (1/76)0.05% (1/1911)0.31% (6/1911)
Implicit99.05%1.72% (34/1981)1.70% (34/2000)3.03% (2/66)0.10% (2/1911)0.31% (6/1911)
MoshiExplicit81.50%1.41% (23/1630)1.15% (23/2000)19.00% (42/221)2.20% (42/1911)3.56% (68/1911)
Implicit82.20%1.58% (26/1644)1.30% (26/2000)21.16% (51/241)2.67% (51/1911)4.66% (89/1911)
FLM-AudioExplicit95.05%0.11% (2/1901)0.10% (2/2000)0.82% (8/977)0.42% (8/1911)0.58% (11/1911)
Implicit95.45%0.05% (1/1909)0.05% (1/2000)0.59% (6/1018)0.31% (6/1911)0.47% (9/1911)
Freeze-OmniExplicit4.25%0.00% (0/85)0.00% (0/2000)6.45% (6/93)0.31% (6/1911)0.31% (6/1911)
Implicit4.55%0.00% (0/91)0.00% (0/2000)2.30% (2/87)0.10% (2/1911)0.16% (3/1911)

Key takeaway: After fixing Voila's streaming codec state, its eligible N4 denominator increases from 620 to 1,029 (Explicit) / 1,032 (Implicit), and its raw count of successful yields (934 / 926) is virtually identical to MiniCPM-o's (935 / 925, or 48.9% vs. 48.9% of all 1,911 N4 events). Reporting both conditional and fixed-denominator views makes cross-model comparison completely transparent.

Reviewer_qsoEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Duplex-MPE evaluates full-duplex models in multi-party conversations, including when to respond, remain silent, and stop speaking. It introduces a matching evaluation protocol and analyzes failures in current open-source models.

Critical strengths
  • Multi-party conversation is an important gap in full-duplex evaluation. The benchmark makes addressee recognition and selective response easier to study.
  • The evaluation separates different interaction behaviors and gives a useful, detailed picture of where current models fail.
Critical weaknesses
  • The data rely on scripted text and synthesized speech. Real multi-party conversations have more complex overlap, timing, prosody, and social cues, so it is unclear whether the findings transfer to realistic settings.
  • Although the evaluation is streaming, the human-side content does not adapt to the model’s responses. This limits what it can tell us about conversational repair and how mistakes affect later turns. Its role alongside dynamic benchmarks also needs clearer explanation.
Questions for authors
  • What does Duplex-MPE measure that dynamic protocols such as TauVoice and FDB-v3 do not? Which parts of the interaction remain scripted?

  • Do the existing sensitivity analyses or any real-audio pilot results support the stability of the model rankings and failure patterns?

4 comments

We thank Reviewer qsoE for the positive assessment of our presentation, problem formulation, and detailed failure analysis. Below we address the two weaknesses and two questions in detail.

Q1 & W2

"What does Duplex-MPE measure that dynamic protocols such as TauVoice and FDB-v3 do not? Which parts of the interaction remain scripted?" "Although the evaluation is streaming, the human-side content does not adapt to the model’s responses. This limits what it can tell us about conversational repair and how mistakes affect later turns. Its role alongside dynamic benchmarks also needs clearer explanation."

1. What Duplex-MPE measures that dynamic dyadic protocols (TauVoice, FDB) do not: Dynamic benchmarks like TauVoice and Full-Duplex-Bench (v1–v3) use an LLM-driven user simulator to evaluate dyadic (1-human + 1-assistant) multi-turn adaptation, conversational repair, and user interruption handling in a 1-on-1 setting. However, they do not evaluate:

  • Multi-party addressee inference (T vs. N1/N2/N3): In a shared room with 3–4 humans, most speech is directed to other humans (N2), other devices (N2), oneself (N3), or merely mentions the assistant in the third person (N1). The assistant must continuously monitor all speakers without a fixed user role.
  • Cross-speaker information integration: Answering T requires combining facts contributed by multiple humans during earlier N2 turns where the assistant was required to stay silent.
  • Third-party floor resolution (N4): Stopping when another human participant steps in to answer or cancel a question just asked to the assistant.
  • Controlled counterfactual addressing pairs: Evaluating the exact same scenario and gold answer under explicit vs. implicit addressing.

2. Why human wording is scripted in Duplex-MPE, and which parts are dynamically reactive vs. scripted: In a 4-party conversation, if the 3–4 human participants were driven by an unconstrained live simulator that dynamically reacted to every model utterance, any false intrusion by a speech model during Turn 1 or Turn 2 (which occurs in 40%–96% of windows for 5 of our 8 models) would derail the human conversation. Once the human speakers stop discussing the venue prices to react to the assistant's intrusion, the subsequent T request and N4 event can no longer occur on comparable evidence, destroying both cross-model comparability and the 2,000 explicit/implicit counterfactual pairs. Therefore, Duplex-MPE uses a hybrid reactive-timing / fixed-script protocol:

  • Dynamically reactive to the model: (i) The model runs in continuous streaming full-duplex mode throughout the ~100 s episode; (ii) After T, the scheduler dynamically adapts its waiting window to the model's real-time speech onset (up to 5 s) and waits for 2 s of confirmed silence (up to 60 s) before the next human turn begins.
  • Scripted across models: The human utterances, speaker order, inter-turn gaps (0.5 s) during N1–N3, and the 3 s answering window between N4_Q and N4_R are held fixed so every model faces identical conversational evidence and interruption timing. We have clarified this complementary relationship with dynamic benchmarks in Sec. 2 and Sec. 3.4 of the revised manuscript.

Q2 & W1

"Do the existing sensitivity analyses or any real-audio pilot results support the stability of the model rankings and failure patterns?" "The data rely on scripted text and synthesized speech. Real multi-party conversations have more complex overlap, timing, prosody, and social cues, so it is unclear whether the findings transfer to realistic settings."

Yes—multiple sensitivity analyses and our expansion to 8 models confirm that the failure patterns and capability rankings are highly stable:

  1. Expansion to 8 models & adapter repairs (Table 3): Across all 8 models and 4,000 streams, the four distinct behavioural archetypes—(i) selective & accurate (MiniCPM-o 4.5, and to a lesser degree Realtime-Venus-Audio), (ii) selective & responsive at yielding but low-content (Voila), (iii) floor-holding / indiscriminate speaking (PersonaPlex, Moshika-RL, Moshi, FLM-Audio), and (iv) over-conservative silence (repaired Freeze-Omni)—remain sharp and consistent across explicit and implicit conditions.
  2. Timing deadline sweeps (1 s–5 s for T; 0.5 s–3.0 s for N4): Sweeping the T response deadline from 2 s2\text{ s} to 5 s5\text{ s} and the N4R\text{N4}_R stopping deadline from 1.0 s1.0\text{ s} to 3.0 s3.0\text{ s} preserves the exact capability tiers across all 8 models:
ModelT Onset Lat. (Med / p90)Fresh-Onset @ 1.0s@ 2.0s@ 3.0s@ 5.0s (Paper)N4 Onset Lat. (Med / p90)AY @ 1.0s@ 1.5s@ 2.0s (Paper)@ 3.0s
MiniCPM-o 4.51.73s / 2.27s2.70%70.62%94.17%94.40%1.78s / 2.33s18.30%34.94%56.07%84.26%
Realtime-Venus-Audio1.54s / 2.40s14.27%64.30%81.65%83.70%1.62s / 2.39s11.85%22.41%36.55%60.78%
Moshika-RL-Seamless0.16s / 0.64s83.47%87.65%88.45%89.28%0.15s / 0.28s1.53%4.15%5.02%8.73%
Voila1.31s / 1.93s1.75%73.12%78.15%80.70%1.30s / 1.69s79.77%86.41%90.25%94.42%
PersonaPlex0.42s / 0.48s76.98%76.98%76.98%76.98%0.41s / 0.47s1.41%1.41%2.11%2.82%
Moshi0.16s / 1.63s56.20%60.42%63.15%65.18%0.15s / 0.74s6.28%12.99%20.13%28.79%
FLM-Audio1.15s / 1.31s5.05%53.23%53.65%54.10%1.14s / 1.27s0.25%0.45%0.70%1.20%
Freeze-Omni0.38s / 1.68s1.05%1.10%1.20%1.23%0.50s / 0.73s3.33%3.89%4.44%6.11%
  1. Acoustic duration sweeps vs. semantic backchannel exemptions: Even when removing the LLM judge completely on N1–N3 windows and scoring silence preservation purely by acoustic VAD duration thresholds τ∈{0,100,200,300,500} ms\tau \in \{0, 100, 200, 300, 500\}\text{ ms}, the separation between selective models (MiniCPM-o, Voila, Freeze-Omni: >76>76%), intermediate models (Venus: ∼60\sim 60%), and indiscriminate speakers (FLM-Audio, Moshi, Moshika-RL, PersonaPlex: <40<40%) remains intact.
  2. Scenario-cluster bootstrap & cross-judge stability: 10,000-replicate scenario-level bootstrap 95% confidence intervals (App. A7.1) confirm significant separation across all 28 model pairs on SP, and cross-family judging with Doubao-Seed-2.1-Pro (98% agreement with Opus 5) leaves model rankings unchanged.
AuthorsOpenReview ↗

Polite follow-up on our responses and revised manuscript

Dear Reviewer qsoE,

As the author–reviewer discussion period progresses, we wanted to politely check in to see whether our responses and revised manuscript have addressed your concerns, and whether you have any remaining or new questions.

Please let us know if our responses have resolved your concerns or if there is anything else we can clarify during the discussion period. Thank you again for your time and thoughtful review!

Best regards.

Dear Reviewer qsoE,

With the discussion deadline approaching, we want to leave a brief final note thanking you for your constructive review.

In your review, you noted that your soundness concerns were minor or fixable (Soundness: 2) and asked how Duplex-MPE complements dynamic protocols (Q1 & W2) and whether sensitivity analyses or more realistic audio pilots support the stability of our findings (Q2 & W1). In the revised manuscript, we have:

  1. Clarified our hybrid reactive-timing / fixed-script design alongside dynamic dyadic benchmarks such as TauVoice and FDB-v3 (Sec. 2 & Sec. 3.4);
  2. Expanded evaluation from 5 to 8 full-duplex models (4,000 streams each) and added onset/stopping deadline sweeps (Appendix Table 24), VAD duration sweeps, and cross-family judge validation (Doubao-Seed-2.1-Pro, 98% agreement), confirming complete stability of the capability tiers;
  3. Added new 8-model experiments with female, child, and elderly voices + human-human overlapping speech (739 latching overlaps + 829 backchannel overlaps, Appendix Table 26), showing that diverse voices preserve model rankings while human-human overlap further amplifies false intrusions and missed turn boundaries.

If you have a brief moment before the discussion window closes, we would be sincerely grateful if you could take a look at these revisions and consider whether they resolve your concerns in your final assessment of our paper. Thank you again for your valuable feedback!

Best regards

Reviewer_4PYkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Duplex-MPE, a benchmark that tests whether full-duplex speech models can act as an assistant in conversations. The model tells when someone is speaking to it, uses context from the surrounding conversation, and stays quiet when it is not addressed. The benchmark has 2,000 synthetic conversations, written by an LLM and voiced with TTS. The authors test five open-weight speech models, which mostly fail the task.

Critical strengths
  • The paper studies an important and underexplored problem. Existing full-duplex benchmarks mostly assume one designated user. Real assistants increasingly work in shared spaces where several people talk and only some of what they say is meant for the assistant.

  • The evaluation is well thought out. The metrics separate different kinds of behavior that one overall score would hide, and the explicit and implicit versions of each scenario make paired comparisons possible.

  • The data pipeline is automatic and scalable, and the authors plan to release it.

Critical weaknesses
  1. The benchmark separates models poorly. Most systems score close to zero, so it is unclear whether they fail at multi-party interaction or at more basic things like QA. A single-speaker control and a check that the harness reproduces each model's known behavior would help separate these causes. Only one model gives a meaningful signal.

  2. The explicit and implicit comparison is confounded. The instruction tells the model to respond only when spoken to by name. That alone could explain the text model's large gap. Most requests not meant for Aria also name another person, so a simple heuristic could do well. The speech models show no gap, but that says little because they respond almost everywhere.

  3. The metric for stopping when a human takes over does not isolate the intended skill. A model that stops whenever anyone speaks gets full credit. The set of scored cases also varies a lot across models, so the numbers are hard to compare. The paper needs a control where a human interrupts without resolving the request.

  4. The audio is very different from real use. All speech is clean TTS from a few fixed voices, with no overlapping speech or background noise. It is unclear whether the conclusions would carry over to real meetings or homes.

  5. A cascaded baseline is missing. A pipeline built from speech recognition, speaker separation and an LLM would show whether the task can be solved from audio at all. It would also separate acoustic difficulty from reasoning difficulty.

  6. The answer evaluation depends heavily on one LLM. The same model writes the reference answers and judges correctness, and the human check covers only a small sample.

Questions for authors

See Weakness.

7 comments

We thank Reviewer 4PYk for the thoughtful review and for highlighting the importance of multi-party evaluation, our decomposed metric design, and our scalable pipeline. Below we address each point.

W1

"The benchmark separates models poorly. Most systems score close to zero, so it is unclear whether they fail at multi-party interaction or at more basic things like QA. A single-speaker control and a check that the harness reproduces each model's known behavior would help separate these causes. Only one model gives a meaningful signal."

1. Expanding to 8 models establishes a clear performance gradient across all metrics. In the revised manuscript, we expanded our evaluation from 5 to 8 open-weight full-duplex models by adding Realtime-Venus-Audio, Moshika-RL-Seamless, and PersonaPlex (Table 3 in the revised paper). Rather than only one model providing signal on AA, Realtime-Venus-Audio achieves 23.44% (Explicit) and 23.18% (Implicit) conditional answer accuracy, forming a clear intermediate tier between MiniCPM-o 4.5 (45.29% / 46.45%) and the remaining six models (≤2.21\le 2.21%). Across all four scored capabilities (Fresh-onset, AA, SP, and AY), the 8 models now exhibit a well-spread performance gradient rather than clustering together, clearly separating distinct capability profiles (please refer to Table 3 in the revised paper for detailed numbers).

2. Harness & adapter verification. Following your suggestion, we audited our evaluation harness and model adapters against each model's official streaming implementation and fixed issues present in the initial submission (most notably in Freeze-Omni and Voila). After fixing these harness/adapter issues, Freeze-Omni's SP improves from 0.00020.0002 to 0.8249/0.82110.8249 / 0.8211 and Voila's fresh-onset rate improves from 0.64650.6465 to 0.80150.8015 (AY to 0.90770.9077), confirming that our updated harness faithfully reproduces each model's native streaming behavior.

3. Why do several models perform reasonably on turn-taking metrics yet score <2.5<2.5% on AA? Basic turn-taking (such as initiating speech when a human stops talking, or staying silent while a human is speaking) is a relatively low-level acoustic/prosodic skill: a model can achieve high response presence, fresh-onset, or even stopping rates simply by reacting to voice activity and pauses—speaking when humans pause and pausing when humans speak. By contrast, Conditional Answer Accuracy (AA) requires the model to genuinely comprehend the question and track factual information across a long (~100 s) multi-speaker context. Consequently, a model can perform well on timing and turn-taking metrics while scoring poorly on AA. Exposing this exact discrepancy between surface turn-taking fluency and actual multi-party comprehension is precisely why we designed Duplex-MPE with separate, decomposed metrics rather than a single aggregate score.

4. New Single-Speaker Control experiments across all 8 models (Appendix Table 25). To directly disentangle foundational spoken QA ability from multi-turn and multi-party interaction challenges, we sampled a stratified subset of N=100N=100 scenarios and redesigned each scenario's spoken content into two genuine single-speaker controls (where a single human speaker talks in the room with the exact same factual premises, question, and reference gold answer, evaluated under the exact same streaming adapters, duty preamble, Silero VAD, Qwen3-ASR, and LLM judge):

  • Single-Speaker 1-Turn Control (1-Turn SS): A single speaker states all relevant facts/constraints and asks Aria the question in one continuous turn (T), isolating foundational spoken QA ability.
  • Single-Speaker Multi-Turn Monologue Control (6-Turn SS): A single speaker talks across 6 turns (N3 self-talk stating part of the facts →\rightarrow N2 device command →\rightarrow N3 self-talk stating remaining facts →\rightarrow N4 self-resolved side question →\rightarrow T final question to Aria), isolating multi-turn streaming context and non-target turns from multi-speaker cross-talk.
ModelFresh-Onset: Multi-PartyFresh-Onset: 6-Turn SSFresh-Onset: 1-Turn SSCond. AA: Multi-PartyCond. AA: 6-Turn SSCond. AA: 1-Turn SS (Δ\Delta vs. MP)Uncond. AA: MP / 6-Turn / 1-Turn SS
MiniCPM-o 4.597.0%95.0%100.0%48.5%52.1%68.0% (+19.5 pp)47.0% / 50.0% / 68.0%
Realtime-Venus-Audio86.0%93.0%98.0%21.3%28.1%47.0% (+25.7 pp)20.0% / 27.0% / 47.0%
Voila77.0%89.0%96.0%1.3%2.2%13.5% (+12.2 pp)1.0% / 2.0% / 13.0%
Moshika-RL-Seamless89.0%92.0%93.0%4.0%4.0%9.1% (+5.1 pp)4.0% / 4.0% / 9.0%
Moshi71.0%83.0%75.0%1.1%5.4%6.3% (+5.2 pp)1.0% / 5.0% / 6.0%
PersonaPlex75.0%85.0%73.0%2.1%3.0%5.0% (+2.9 pp)2.0% / 3.0% / 5.0%
FLM-Audio53.0%27.0%61.0%0.0%0.0%0.0% (0.0 pp)0.0% / 0.0% / 0.0%
Freeze-Omni1.0%6.0%0.0%0.0%0.0%0.0% (0.0 pp)0.0% / 0.0% / 0.0%

These results cleanly separate the two causes of low AA: (i) stronger omni models possess solid foundational spoken QA capability (68.0% for MiniCPM-o 4.5 and 47.0% for Realtime-Venus-Audio in 1-Turn SS), and their drop in Duplex-MPE (-19.5 pp and -25.7 pp) comes from accumulating facts across streaming non-target turns (6-Turn SS: 52.1% and 28.1%) and multi-party cross-talk (Multi-Party: 48.5% and 21.3%); (ii) smaller native speech-to-speech models (Voila 13.5%, Moshika-RL 9.1%, Moshi 6.3%, PersonaPlex 5.0%) already face a foundational spoken QA bottleneck on arithmetic/constraint reasoning in 1-Turn SS, which multi-party interaction then degrades by a further 2$\times$–10$\times$. We have added this complete analysis to Appendix Table 25 of the revised paper.

W2

"The explicit and implicit comparison is confounded. The instruction tells the model to respond only when spoken to by name. That alone could explain the text model's large gap. Most requests not meant for Aria also name another person, so a simple heuristic could do well. The speech models show no gap, but that says little because they respond almost everywhere."

We respectfully clarify three points supported by our dataset design and empirical controls:

  1. The duty instruction does NOT instruct the model to respond only when spoken to by name. First, Gemini 3.1 Pro still responds to 30.40% of implicit T requests where the name "Aria" never appears (608 / 2,000 requests), directly showing that the instruction does not restrict responses to turns containing the name. Second, the verbatim instruction (App. A3.1) is: "Only respond when someone speaks directly to you, Aria, and asks something you can answer." Here, "to you, Aria" appositionally identifies the assistant's persona/role (introduced in the preceding sentence: "You are the voice assistant Aria..."); it specifies direct addressing, not a lexical wake-word requirement, and our human validation (Table 11) confirmed that 100% (50/50) of implicit T turns are unambiguously addressed to Aria through context.
  2. A simple wake-word heuristic (responding whenever "Aria" is mentioned) fails on Duplex-MPE. Every scenario includes N1 turns (3,041 explicit / 3,039 implicit windows) that explicitly mention the name "Aria" (e.g., "See, this is exactly why we keep Aria in here") yet require silence. A heuristic that responds whenever "Aria" is mentioned fails on 100% of N1 turns.
  3. Speech models do NOT "respond almost everywhere"—MiniCPM-o 4.5 and Voila are highly selective. While PersonaPlex and Moshi speak indiscriminately, MiniCPM-o 4.5 preserves silence on 92.67% / 92.83% of all 20,360 N1–N3 windows, and Voila preserves silence on 83.97% / 84.27%. Specifically, on the 4,704 non-addressed questions/requests in N1/N2 (2,570 directed to another device and 2,134 directed to another human, Table 19), MiniCPM-o's false-response rate is only 3.95% (4.7% on device requests, 3.0% on human requests), and on N3 self-talk its false-response rate is only 4.9%. Yet on T requests addressed to Aria, MiniCPM-o responds 95.50% (Explicit) and 94.30% (Implicit) of the time (Δ=+1.20 pp,p=0.0792\Delta = +1.20\text{ pp}, p = 0.0792). Thus, MiniCPM-o genuinely distinguishes addressed requests (both explicit and implicit) from unaddressed speech.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response (To Area Chair and All Reviewers)

We thank the Area Chair and all three reviewers (4PYk, qsoE, HBEH) for their constructive feedback. We are encouraged that all reviewers recognize the importance of extending full-duplex speech evaluation beyond single-user dyadic exchanges to shared multi-party conversations, and appreciate our 4-capability metric decomposition for exposing distinct interaction failure modes that single response-rate metrics conceal.

To thoroughly address the reviewers' comments, we have conducted expanded evaluations, interface audits, and targeted control analyses, and prepared a revised manuscript:

  1. Expanded 8-Model Evaluation & Adapter Verification (Addressing 4PYk-W1, HBEH-Q2, qsoE-Q2):
    • We expanded evaluation from 5 to 8 open-weight full-duplex speech models by adding Realtime-Venus-Audio, Moshika-RL-Seamless, and PersonaPlex across all 4,000 streams (2,000 pairs). Notably, Realtime-Venus-Audio achieves 23.44% / 23.18% conditional answer accuracy (AA), 60.32% / 60.87% silence preservation (SP), and 36.30% / 36.79% answering-window yield (AY), establishing a clear intermediate performance tier between MiniCPM-o 4.5 (~45.3%–46.5% AA) and the lower-scoring models.
    • We also audited and fixed issues in our initial evaluation harness and streaming model adapters against each model's official implementation (HBEH-Q2, 4PYk-W1). For instance, after fixing the Freeze-Omni adapter, its SP improves from 0.0002 to 0.8249 / 0.8211 while its RP drops from 0.9955 to 0.0425 / 0.0455, confirming that its previously reported continuous speech was an adapter issue rather than a model behavior.
ModelCond.Fresh-onset ↑Cond. AA ↑Silence pres. (SP) ↑Answering yield (AY) ↑Response pres. (RP)Window resp. (WR)
MiniCPM-o 4.5Explicit / Implicit0.9480 / 0.94000.4529 / 0.46450.9267 / 0.92830.5629 / 0.55860.9550 / 0.94300.9513 / 0.9498
Realtime-Venus-AudioExplicit / Implicit0.8320 / 0.84200.2344 / 0.23180.6032 / 0.60870.3630 / 0.36790.9110 / 0.91900.9231 / 0.9309
VoilaExplicit / Implicit0.8015 / 0.81250.0124 / 0.01650.8397 / 0.84270.9077 / 0.89730.8050 / 0.81700.7823 / 0.7849
Moshika-RL-SeamlessExplicit / Implicit0.8775 / 0.90800.0196 / 0.02210.4753 / 0.48030.0556 / 0.04550.9930 / 0.99401.0000 / 1.0000
PersonaPlexExplicit / Implicit0.7555 / 0.78400.0217 / 0.01720.0413 / 0.04000.0132 / 0.03030.9895 / 0.99051.0000 / 1.0000
MoshiExplicit / Implicit0.6340 / 0.66950.0141 / 0.01580.2753 / 0.27050.1900 / 0.21160.8150 / 0.82200.9231 / 0.9309
FLM-AudioExplicit / Implicit0.5385 / 0.54350.0011 / 0.00050.3963 / 0.39940.0082 / 0.00590.9505 / 0.95450.9927 / 0.9942
Freeze-OmniExplicit / Implicit0.0145 / 0.01000.0000 / 0.00000.8249 / 0.82110.0645 / 0.02300.0425 / 0.04550.1805 / 0.1795
  1. Fixed-Denominator (Unconditional) Reporting Alongside Conditional Metrics (Addressing HBEH-W2, 4PYk-W3):

    • To enable direct comparison on identical denominators across all models, we report unconditional rates keeping the same numerators while fixing the denominators to all 2,000 T requests and all 1,911 N4 events: Unconditional AA (/2,000 T) evaluates a model's joint ability to both respond to a request and answer it correctly, while Unconditional AY (/1,911 N4) evaluates a model's joint ability to both initiate a timely response after N4Q\text{N4}_Q and stop in time when N4R\text{N4}_R begins. Under fixed denominators, model rankings are completely preserved: e.g., for Unconditional AA (/2,000 T), MiniCPM-o achieves 43.25% / 43.80%, Venus achieves 21.35% / 21.30%, and all other six models remain ≤\le 2.20%; for Unconditional AY (/1,911 N4), MiniCPM-o (48.93% / 48.40%) and Voila (48.87% / 48.46%) tie at the top.
  2. Strict Paired Controls & Selectivity Evidence for Explicit vs. Implicit Addressing (Addressing 4PYk-W2, HBEH-W3, HBEH-Q1):

    • We isolate the 1,661 strictly T-only pairs (where every turn except T has identical text): Gemini 3.1 Pro's explicit–implicit gap remains +63.64 pp (p<10−304p < 10^{-304}), while all 8 speech models show ∣ΔRP∣≤1.87 pp|\Delta\text{RP}| \le 1.87\text{ pp} (p≥0.0987p \ge 0.0987).
    • We further show from Table 19 that MiniCPM-o is not indiscriminately answering every question: while responding to 95.5% / 94.3% of explicit/implicit T requests, it stays silent on 97.0%–98.0% of the 2,134 questions directed to other humans (N1/N2), 95.0%–95.3% of the 2,570 requests to other devices (N2), and 94.7%–95.1% of the 3,219 self-talk turns (N3).
  1. Timing Deadline Sensitivity & Cross-Family LLM Judge Validation (Addressing HBEH-Q3, 4PYk-W6):

    • Sweeping the T onset deadline across 1,2,3,5 s1, 2, 3, 5\text{ s} and the N4R\text{N4}_R stopping deadline across 0.5,1.0,1.5,2.0,3.0 s0.5, 1.0, 1.5, 2.0, 3.0\text{ s} confirms that all 8 models' median onset latencies (0.16 s0.16\text{ s}–1.73 s1.73\text{ s}) and 90th-percentile latencies (0.48 s0.48\text{ s}–2.40 s2.40\text{ s}) lie well below 3 s3\text{ s}, and model rankings are invariant across reasonable thresholds.
    • Beyond our 6-annotator human validation (99% agreement with Claude Opus 5 on T answer accuracy and 98% on silence preservation), we conducted a cross-family LLM judge evaluation using Doubao-Seed-2.1-Pro, which achieves 98% agreement with Claude Opus 5.
  2. New Single-Speaker Controls & Diverse-Voice / Overlapping-Speech Pilots (Addressing 4PYk-W1, HBEH-W1):

    • Single-Speaker Control (4PYk-W1, Appendix Table 25): On a stratified 100-scenario subset, we rewrote each scenario into a natural 1-Turn Direct Single-Speaker request and a Multi-Turn Single-Speaker Monologue (with identical underlying facts, questions, gold answers, streaming adapters, preamble, VAD, ASR, and judge). Moving from multi-party interaction to 1-turn single-speaker QA boosts MiniCPM-o 4.5 AA from 48.5% to 68.0% (+19.5 pp), Realtime-Venus-Audio from 21.3% to 47.0% (+25.7 pp), Voila from 1.3% to 13.5% (+12.2 pp), and Moshika-RL / Moshi / PersonaPlex from 1.1%–4.0% to 5.0%–9.1%, cleanly separating foundational spoken QA capability from multi-turn and multi-party degradation.
    • Female, Child, and Elderly Voices + Human-Human Overlap (HBEH-W1, Appendix Table 26): Re-synthesizing the 100-scenario subset (1,297 turns) with young female, adult female, 10-year-old child, and 70-year-old elderly voices across all 8 models yields highly consistent capability rankings (MiniCPM-o 4.5: 92.0% Fresh-Onset, 43.0% AA, 93.6% SP, 58.2% AY; Realtime-Venus-Audio: 89.0% Fresh-Onset, 21.1% AA vs. 21.3% baseline, 68.3% SP), while further injecting human-human turn-transition overlaps (350 ms) and backchannel overlaps degrades turn-boundary detection and silence preservation across selective models (MiniCPM-o 4.5 Fresh-Onset 82.0%, SP 70.3%, AA 43.9%; Voila Fresh-Onset 72.0%, SP 72.2%; Venus AA 15.1%, SP 62.0%), confirming that acoustic overlap further exacerbates the turn-taking and intrusion challenges exposed by our clean baseline.
AuthorsOpenReview ↗

Thank you for your feedback and engagement

Dear Reviewer 4PYk,

Thank you very much for taking the time to read our responses and engage in the discussion! We are glad to hear that our rebuttal addressed most of your questions.

Since you mentioned that most of your questions were addressed, if there is any remaining concern or specific point you would like us to clarify further while the discussion window is open, please let us know, we would be very happy to address it. Thank you again for your constructive feedback, which has substantially strengthened the paper!

Best regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.