acceptodds
Under review as a conference paper at ICLR 2027

Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models

Abstract

A full-duplex model should answer easy questions immediately, but hard questions often need stronger reasoning, which may require waiting. The challenge is to choose between these paths at the moment the model begins to speak. Recent systems learn related mode decisions through special-token generation inside the dialogue model, coupling routing to turn-taking and temporally annotated fullduplex training. We study a post-training alternative. We freeze MiniCPM-o 4.5, using only its speech input and output, and fit a logistic failure readout on hidden states already available at its native listen-to-speak transition. The resulting gate requires no additional model generation and exposes thresholds that can be retuned without changing the duplex policy. A probe fitted on text-input states also reads the states of spoken questions, including unseen human recordings. On the internal benchmark, selective handoff improves answer accuracy by 27.1 points. Compared with always using the expert, it reduces mean server time to first answer audio by 31.5 percent. It also outperforms matched random routing across external speech question-answering pools, as well as a zero-shot GPT-5.4 mini router that rates the transcribed question. The training recipe also transfers to a second duplex model family. These results show that an existing duplex model can be equipped to answer easy turns immediately and wait for stronger reasoning or fresher knowledge on the turns that need it, without further duplex training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
full-duplex speech models, hidden-state probing, failure prediction, selective escalation, model routing, conversational agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XcXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-training expert-routing framework for real-time duplex speech models. A lightweight logistic regression probe uses hidden states at the native listen-to-speak boundary to predict local answer failures and selectively escalate high-risk queries to a stronger expert model, without fine-tuning the speech model or requiring additional language-model generation. Experiments on an internal benchmark and four external speech-QA datasets demonstrate improved accuracy over local-only and matched-random routing, while reducing server waiting time compared with always escalating. Cross-modal transfer experiments, evaluation on a second model family, and a small user study further support the approach's potential. However, its effectiveness varies across failure types, and broader validation in real-world settings remains necessary.

Critical strengths
  1. The method uses hidden states at the native listen-to-speak boundary to enable selective expert routing without modifying or fine-tuning the duplex model, making it easy to integrate and adjust at deployment time.

  2. Experiments across internal and external speech-QA benchmarks show consistent gains over local-only and matched-random routing, while reducing server waiting time compared with always escalating. Cross-modal transfer and evaluation on a second model family further support its potential generalizability.

Critical weaknesses
  1. The probe performs poorly on multi-step reasoning errors (AUC 0.47), raising concerns about its ability to identify important failure cases. This limits the reliability and general applicability of the proposed routing framework.

  2. The user study includes only 13 participants and 16 task pairs, while the latency evaluation does not fully capture end-to-end user-perceived latency. These limitations weaken the evidence for practical deployment benefits and leave the claimed real-time advantages insufficiently established.

Questions for authors
  1. Given the near-random AUC (0.47) for multi-step reasoning errors, how would the authors characterize the current method’s limitations in detecting reasoning failures?

  2. Since TTFA excludes endpoint-detection and client playback delays, how representative are the reported latency gains of actual user-perceived latency in live deployment?

  3. How sensitive are the probe and routing thresholds to distribution shifts across languages, tasks, and unseen speech inputs, given the varying escalation rates across external datasets?

  4. Could the authors clarify whether a trained or calibrated text-based router would narrow the gap with the proposed hidden-state probe, beyond the zero-shot GPT-5.4 mini baseline?

  5. Given the small user study (13 participants, 16 task pairs), how robust are the reported preference differences, particularly for simple factual tasks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer XcXk (Part 1/5)

We thank you for the feedback. We address your questions below.

Q1: If execution errors are hard to detect, what value does the probe provide?

The probe directs expert calls to questions that need more help. In this comparison at the same expert-call budget, it corrects more errors than random escalation, including some execution errors.

  • Execution errors are not the main source of failure: These occur when the model understands the question but makes a mistake in calculation, counting, or a reasoning step. In this set of failures, confident, specific but incorrect answers are more common, and they account for most of the probe's error-recovery benefit.

  • A single decision can still correct some execution errors: With local and expert answers fixed, the probe uses the same number of expert calls as random escalation and corrects more execution errors. It also corrects more confident but incorrect answers.

Additional experiment: Building on the existing error-type analysis, we added random escalation at the same expert-call budget and compared error-recovery rates with fixed answers.

Error types among 600 incorrect local answers

Error type (model-labeled)CountShare of all errors
Execution errors447.3%
Confident but incorrect answers41869.7%

Fixed-answer replay: Error recovery at the same budget

Both routing methods call the expert on 687 of 1,392 questions (49.4%).

MethodExecution-error recoveryRecovery of confident errors
No expert0%0%
Random escalation (expected)29.0%41.2%
Probe routing36.4%62.0%
AuthorsOpenReview ↗

Response to Reviewer XcXk (Part 2/5)

Q2: Does the original TTFA reflect the user's actual wait?

The original TTFA measures only when the server has prepared the answer. The added client-side TTFA includes audio transmission and playback scheduling, making it closer to the user's actual wait.

  • Timing to the start of a substantive answer: We use a single client clock to measure from the end of the user's final speech segment to client-side scheduling of the first substantive answer audio. Waiting prompts such as "Please hold on" do not count as an answer.

  • The long tail mainly comes from waiting for the expert: Session logs show that all of the slowest answers in this batch used the expert path, with most of the time spent waiting for the expert to return. We therefore report local and expert-answer waiting times separately.

  • Answered and unanswered sessions are both accounted for: Overall results cover both local and expert paths. Sessions without a substantive answer are counted separately rather than treated as fast responses.

Additional analysis: We reviewed existing session logs, split client-side TTFA by the actual answer source, and checked the timelines of long-tail sessions.

Client-side TTFA: Local answers and expert handoff (240 questions)

Answer sourceAnsweredMean (s)Median (s)
Local answer1183.053.05
Expert handoff11910.248.85
Overall2376.665.54
AuthorsOpenReview ↗

Response to Reviewer XcXk (Part 3/5)

Q3: Does the probe work on datasets it did not see during training?

Yes. After training on the original data, the probe is kept fixed and applied directly to unseen test sets, where it still distinguishes questions that are more likely to be answered incorrectly.

  • The decision method is set using the original data: The original calibration set is used to train the probe and select the layer and internal features. These choices are finalized before the resulting probe is evaluated on the subsequent test sets.

  • No retraining on the new test sets: We fix the probe's weights, layer, and features and directly evaluate a separate set of Chinese questions. These questions were deduplicated against existing data and were not used to fit the probe, so the test evaluates transfer to unseen data.

  • Failure risk remains distinguishable on unseen tests: Across factual QA, knowledge exams, and causal commonsense tasks, the fixed probe ranks questions that are more likely to be answered incorrectly ahead of others. Its decision signal therefore transfers to these datasets that were not used in training.

Additional reporting: These experiments were completed before submission. We now clarify the training-test separation and present the unseen-test results first.

Unseen Chinese test sets: Direct evaluation with a frozen probe

DatasetAUC
Chinese SimpleQA0.719
C-Eval0.730
CMMLU0.642
XCOPA-zh0.693
Equal-weight task mean0.696

External English tasks: Additional results with a fixed probe

The same probe still distinguishes questions that are more likely to be answered incorrectly on these new test sets. Different proportions of questions trigger escalation across datasets, so expert-call rates also differ.

DatasetAUCExpert-call rate (%)
Speech TriviaQA0.79755.2
Speech WebQ0.77956.8
Speech Llama0.74421.2
SD-QA0.81262.5
Reviewer_sQYgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates expert routing for full-duplex speech interaction models through post-training. With the backbone frozen, a linear probing head is trained to determine whether a more capable expert model should be invoked to answer a given question. The authors explore different routing strategies, revealing a trade-off between response accuracy and latency. They further investigate the cross-modal transferability of failure probes and analyze the properties of intermediate representations across different model layers.

Critical strengths

The idea of leveraging a more capable expert model to assist full-duplex speech interaction systems in answering challenging questions is interesting. he experimental results demonstrate noticeable improvements in question-answering accuracy.

Critical weaknesses

My primary concern is whether the proposed expert routing framework can preserve the fundamental advantages of full-duplex interaction. Although aggressive expert routing leads to promising improvements in question-answering accuracy, it also introduces substantial additional latency. This raises an important question: can the proposed framework still achieve lower end-to-end latency than conventional cascaded ASR+LLM+TTS systems?

If the expert routing framework exhibits both higher latency and lower question-answering accuracy than a cascaded system, its practical advantages become questionable, particularly in scenarios where response accuracy is the primary objective. In such cases, it remains unclear why one would prefer an expert-augmented full-duplex model over a conventional cascaded architecture.

Moreover, if the motivation is to preserve the naturalness and responsiveness of full-duplex interaction, the impact of expert routing on conversational quality should be carefully evaluated. In particular, frequent expert invocation may introduce noticeable delays, disrupt turn-taking, and lead to fragmented conversational experiences. The current evaluation does not sufficiently establish whether the proposed framework can maintain smooth, natural and interruption-aware interactions while benefiting from improved reasoning capabilities.

Questions for authors
  1. Could the authors evaluate the proposed framework on established full-duplex interaction benchmarks, such as Full-Duplex-Bench, as well as realistic daily conversational scenarios?

  2. Could the authors provide comparisons with cascaded ASR+LLM+TTS systems in terms of both question-answering accuracy and end-to-end interaction latency?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sQYg (Part 1/3)

We thank you for the feedback. We address your questions below.

Q1: Why retain a local full-duplex model instead of using a cascade?

We retain the local full-duplex model so the system can keep listening while speaking, decide when to respond, and stop when interrupted. It calls the expert when greater answering capability is needed.

  • The local model handles immediate interaction: It continuously listens to the user and handles interruptions, while the expert answers questions requiring additional knowledge or reasoning. The probe only decides whether to call the expert, adding stronger answering capability on demand while retaining the local model's interaction handling.

  • Interaction and background reasoning are handled separately: TML's Interaction Models and OpenAI's GPT-Live both use an interaction model to face the user and a background model for more complex tasks. Our probe adds a lightweight decision to this division of work: does the current question warrant expert help?

  • The same-expert comparison supports selective escalation: A cascade runs speech recognition, language-model response generation, and speech synthesis in sequence. With the same expert in both systems, ours achieves similar answer accuracy with fewer expert calls while retaining the local interaction path.

New experiment: We added a cascade using the same expert and interleaved its runs with ours on the same speech questions, measuring answer accuracy, expert-call rate, and client-side waiting time.

Related work: TML Interaction Models; OpenAI GPT-Live

Same expert: Answer quality and expert use

SystemAccuracy (%)AnsweredExpert calls (%)
Ours69.223751.7
Cascade68.320599.6

Paired questions: Accuracy and waiting time

MetricDifference (Cascade - Ours)95% CIPairs
Accuracy (pp)−0.8[−6.2, 4.6]240
TTFA (s)−0.70[−1.14, −0.27]204
AuthorsOpenReview ↗

Response to Reviewer sQYg (Part 2/3)

Q2: How well is the original model's interaction ability retained after adding an expert?

Fast local audio onset and stopping after interruptions are retained: both remain close to the original model. Our method also speaks out of turn less often than the cascade when users pause or speak to someone else.

  • After the user finishes speaking, our method starts producing audio at a speed close to the original model. It also stops the previous answer quickly when the user interrupts. Neither timing measure shows a clear additional delay.

  • When users pause to think or speak to someone else, our method speaks out of turn less often than the cascade and performs close to the original model. Retaining the local full-duplex model lets the system continue using conversational context to decide when to respond.

  • After an interruption, our method still speaks in most tests, with a waiting time close to that of the original model.

New experiments: We completed the FDB comparison of the original model, our method, and the cascade. We also retested audio onset, stopping, and post-interruption responses for the original model and our method using the same runtime version.

The original model is MiniCPM-o 4.5. Our method adds on-demand expert calls to that model. The cascade consists of speech recognition, Qwen3-8B, and speech synthesis.

FDB: Original model, our method, and cascade

Conclusion: Our method speaks out of turn less often than the cascade when users pause or speak to someone else. Across these six scenarios, its results are generally close to those of the original model.

The first five rows measure the rate of speaking when the system should not; lower is better. The last row measures the rate of answering the new question after a genuine interruption; higher is better.

Scenario and measure (%)Original modelOursCascade
Thinking pause (human): unwanted response ↓0.00.529.8
Thinking pause (synthetic): unwanted response ↓0.00.07.3
Background speech: unwanted response ↓15177
Speaking to someone else: unwanted response ↓131791
User backchannel: unwanted response ↓213
Genuine interruption: answers the new question ↑191896

Local responsiveness: Audio onset and stopping (server-side means)

Conclusion: Our method starts producing audio and stops the previous answer after an interruption at speeds close to the original model. Neither measure shows a clear additional delay.

MeasureOriginal modelOursPairs
First audio after user speech ends (including prompts, s)1.3581.342119
Time to stop the previous answer after interruption (s)0.1150.106136

After interruption: Speaking, waiting, and relevance

Conclusion: After an interruption, our method retains a high rate of speaking within the test window, with a waiting time close to that of the original model.

MeasureOriginal modelOurs
Speaking within the time limit (%)95.093.5
Wait when both systems speak (s)3.0143.145
Relevance when both systems speak (out of 5)4.504.16
AuthorsOpenReview ↗

Response to Reviewer sQYg (Part 3/3)

Q3: Is waiting for the expert worthwhile in everyday conversation?

Waiting for the expert is valuable when current information is needed: users preferred expert-assisted answers on these tasks. When no expert is called, local audio onset remains close to the original model.

  • The escalation decision is lightweight: The probe reads internal states already computed by the local model and produces a failure-risk score. It requires neither an additional self-assessment response from the local model nor a separate model call to decide whether to consult the expert.

  • Without an expert call, audio onset remains close to the original model: In these paired non-escalated samples, adding the probe did not show an additional delay in first audio.

  • Current-information tasks show the value of waiting more clearly: These questions require up-to-date information that the local model lacks. With expert support, users gave higher correctness and helpfulness ratings and preferred our answers.

Additional analysis: We reused existing QA and FDB records to separate local and expert paths and check first-audio timing on non-escalated samples.

No expert call: Time to first audio (mean over 10 pairs)

MeasureOriginal modelOurs
First audio after user speech ends (including prompts, s)1.6491.612

Waiting time on the two answer paths

Answer pathAnsweredMedian (s)P95 (s)
Local answer1183.053.76
Expert answer1198.8519.70

Current-information tasks: User ratings (out of 5)

MetricLocal modelSelective escalation
Correctness2.563.67
Helpfulness2.003.67

User preferences: Simple conversation and current information (13 participants)

TaskOurs winsTiesOurs loses
Simple conversation313
Current information630

Our method: Expert use in the user study

TaskEscalated turnsEscalation rate per turn
Simple conversation (S1)9/3129.0%
Current information (S2)19/3751.4%

Thanks the authors for their rebuttal. Although some of my concerns have been addressed, the additional results do not demonstrate a clear performance advantage of the proposed model over conventional cascaded systems. I will maintain my score.

Reviewer_j3SjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a post-training expert-routing framework for full-duplex speech models. It uses a lightweight failure probe on the hidden states of a frozen MiniCPM-o 4.5 to decide whether to answer locally or escalate to a stronger expert. On the internal benchmark, the method improves accuracy from 42.1% to 69.2% while reducing mean server latency by 31.5% relative to always escalating. Additional evaluations cover four external speech-QA benchmarks, a second duplex model, and a small human study.

Critical strengths

1.A practical routing mechanism that preserves the original duplex model. The paper separates the model's decision about when to speak from the decision about whether its answer should be trusted. The failure readout reuses hidden states already available at answer onset, without requiring a separate self-evaluation generation or modifying the underlying duplex policy. This is a useful design for latency-sensitive speech systems. 2.Consistent improvements over meaningful routing baselines. Table 1 shows that the proposed gate improves the external four-benchmark average from 59.4% to 83.2% on MiniCPM-o 4.5, compared with 76.0% for matched random routing. The fixed-outcome replay in Appendix E.1 further supports the benefit of selecting which questions to escalate, rather than merely increasing the overall expert-call rate. These comparisons provide more convincing evidence than an always-local baseline alone. 3.Useful analyses beyond the main accuracy results. The paper examines layer selection, text-to-speech representation transfer, and routing behavior across different model families. The additional VoiceChat-11B experiments and the evaluation on previously unseen human recordings are valuable, although the strength of the generalization claims remains subject to the limitations discussed below.

Critical weaknesses
  1. Insufficient justification for selecting L22 in the deployed audio setting. Section 3.3 motivates L22 primarily using text-input mathematical reasoning results, where its AUC reaches 0.931. However, the deployed system uses audio input and a different feature construction. More importantly, Section 5.2 reports that the final-layer AUC reaches 0.936 under matched audio conditions, suggesting that the late-layer degradation motivating L22 is modality-dependent. A controlled layer comparison using native audio features and matched expert-call budgets is needed to establish whether L22 is an appropriate choice for deployment.
  2. Unexplained discrepancy between the always-expert results across model families. Table 1 reports substantially different always-expert accuracies on the same 240 Internal questions, 79.6% for MiniCPM-o 4.5 and 91.7% for VoiceChat-11B, despite both using GPT-5.5 as the expert. Since evaluation measures answer content rather than synthesized speech quality, this difference cannot be explained simply by the local talkers. Moreover, MiniCPM-o 4.5 has substantially higher local-only accuracy (42.1% versus 19.6%), making the reversed expert results particularly difficult to interpret. The authors should clarify differences in transcription, expert inputs, and evaluation procedures, ideally through a paired comparison with fixed transcripts and expert responses.
  3. Limited evidence of real-world conversational benefits. Section 6 evaluates only 13 participants across 16 paired tasks. Although the gated system improves answer-related ratings, the overall preference results are limited (9 wins, 4 ties, and 3 losses), with little improvement in conversation naturalness (3.06 versus 2.94). These findings provide preliminary evidence but are insufficient to establish general benefits in open-ended, multi-turn duplex conversations, particularly where interruptions and waiting times affect user experience. A larger and more diverse human evaluation would strengthen the practical claims.
Questions for authors

1.Could the authors provide a controlled comparison of probe layers under the native audio input and deployed feature construction, reporting both failure-prediction AUC and routing accuracy at matched expert-call rates? In particular, why is L22 preferred when the late-layer decline largely disappears under matched audio calibration and evaluation? 2.What accounts for the 79.6% versus 91.7% always-expert accuracy on the same Internal query IDs? Were the expert-facing transcripts, expert prompts, inference configurations, and judging procedures identical across the two model families? Could the authors provide a paired fixed-transcript and fixed-expert-response comparison? 3.Can the authors provide stronger evidence of user-facing benefits in open-ended, multi-turn speech conversations? In particular, how robust are the reported preferences across participants and task types, and how do actual audible delays and interruption behavior affect the benefits of selective escalation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer j3Sj (Part 1/3)

We thank you for the feedback. We address your questions below.

Q1: Why choose L22 rather than the final layer?

We chose layer 22 (L22) because its signal is available earlier, its routing performance is close to that of the final layer (L35), and it supports both speech and text input.

  • An earlier decision signal: The features needed to estimate failure risk are available at L22, before computation reaches L35. This creates an opportunity to decide earlier whether an expert is needed and start preparing the expert request.

  • Performance close to the final layer: With the same speech input, feature construction, and expert-call rate, L22 and L35 yield similar final answer accuracy. An intermediate layer therefore retains routing performance close to that of the final layer.

  • Support for speech and text: The same probe can read risk signals from both input modalities. This allows the system to use the same escalation decision method when users switch from speech to text mid-utterance or alternate between the two.

Additional evidence: Profiling of both layers is complete. The L35 comparison was conducted before submission but omitted for space; we now include it.

Higher AUC means better identification of questions the model is likely to answer incorrectly. 95% CI denotes a 95% confidence interval.

Speech evaluation after text-based calibration

Test inputLayerAUC
Speech inputL220.855
Human recordingsL22–L260.760–0.800

Native speech: L22 vs. L35 at a 30% expert-call rate

LayerFailure-detection AUCRouted accuracy (%)
L220.69567.60
L350.66268.13

Profiling: How much earlier is L22?

L22's signal is available earlier, while actual expert-request timing remains close to L35.

EventTime gained (ms)95% CI
Features available9.91[9.89, 9.93]
Expert request14.7[−28.5, 57.8]
AuthorsOpenReview ↗

Response to Reviewer j3Sj (Part 2/3)

Q2: Why do two systems using the same expert have different accuracy?

The correct expert accuracy for the speech-system comparison is 79.6%. Although both evaluations use the same expert and question IDs, the expert receives different inputs: the speech evaluation transcribes the recordings before sending them to the expert, whereas the 91.7% text evaluation supplies the original questions directly. These are different evaluation settings. The original table mistakenly cited the text-evaluation figure; the speech comparison should consistently use 79.6%.

AuthorsOpenReview ↗

Response to Reviewer j3Sj (Part 3/3)

Q3: What does the small user study show about real conversations?

The user study shows that selective expert assistance improves the conversation experience: the clearest benefit is better answers to questions requiring current information, which users also prefer.

  • More correct and helpful answers: On current-information tasks, users gave our system higher ratings for both correctness and helpfulness than the local model. In the preference comparison, users either preferred our system or rated the two equally. The improvement in answer quality was recognized by users.

  • These gains do not require an expert on every turn: Our system escalates only some turns and lets the local model answer the rest. Expert use was more frequent for current-information tasks than for simple conversation. The gains in ratings and preference were achieved with selective expert use.

  • Fast local interaction is retained: In a separate interaction test, our system started producing audio and stopped the old answer after interruptions at speeds close to the original model. Expert assistance therefore preserves the speed of these two local responses.

Current-information tasks: User ratings (out of 5)

MetricLocal modelSelective escalation
Correctness2.563.67
Helpfulness2.003.67

User preferences (13 participants, 16 task pairs)

TaskOurs winsTiesOurs loses
Simple conversation313
Current information630

Our method: Expert use in the user study

TaskEscalated turnsEscalation rate per turn
Simple conversation (S1)9/3129.0%
Current information (S2)19/3751.4%

Overall user study: Mean ratings (out of 5)

MetricLocal modelOurs
Correctness3.254.00
Helpfulness2.883.94
Context consistency3.753.94
Conversational naturalness2.943.06

Measured waiting time: Local and expert answers

Answer pathAnsweredMedian (s)
Local answer1183.05
Expert answer1198.85

Local responsiveness: Audio onset and stopping (server-side means)

Conclusion: Our method starts producing audio and stops the previous answer after an interruption at speeds close to the original model. Neither measure shows a clear additional delay.

MeasureOriginal modelOursPairs
First audio after user speech ends (including prompts, s)1.3581.342119
Time to stop the previous answer after interruption (s)0.1150.106136

After interruption: Speaking, waiting, and relevance

Conclusion: After an interruption, our method retains a high rate of speaking within the test window, with a waiting time close to that of the original model.

MeasureOriginal modelOurs
Speaking within the time limit (%)95.093.5
Wait when both systems speak (s)3.0143.145
Relevance when both systems speak (out of 5)4.504.16

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your thoughtful review. We would like to kindly follow up on our responses and the additional experimental results provided in our rebuttal.

Could you let us know whether these address your main concerns? If any questions remain, we would be happy to provide further clarification.

If you find that the additional evidence has resolved your concerns, we would appreciate it if you could consider revising your score upward, if warranted.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.