acceptodds
Under review as a conference paper at ICLR 2027

CAR-Voice-Bench: Benchmark for In-Car Full-Duplex Voice Assistants

Abstract

In-car voice assistants are integral to modern vehicle control, navigation, and telematics. During natural interactions, users routinely exhibit dynamic conversational behaviors, such as concurrent speech, barge-in interruptions, and abrupt goal modifications. Despite their prevalence, existing automotive benchmarks fail to jointly evaluate full-duplex spoken dialogue, executable tool use, and rigorous outcome verification. To address this deficiency, we propose CAR-Voice-Bench, a unified evaluation framework integrating domain-specific tasks, full-duplex interaction, and a replayable testbed. The benchmark comprises 300 curated tasks covering six practical challenges: driving safety, turn management, tool unavailability, disambiguation, simple, and complex. We systematically evaluate three representative models under both standard and dialectal speech conditions with injected background noise, independently testing each configuration three times. Among them, OpenAI's gpt-realtime-2.1 achieves the best overall performance, attaining a pass@1 reward of 51.61%.

Primary area:
datasets and benchmarks
Keywords:
benchmark, voice assistants

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_8WwDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CAR-Voice-Bench, a benchmark for evaluating full-duplex voice assistants in automotive scenarios. Building upon CAR-bench and τ-Voice, the authors construct 300 Chinese in-car tasks across six categories and develop an evaluation framework combining natural-language (NL), database-state (DB), and action-trace verification. Experiments on three existing voice models examine task completion, tool-use accuracy, and interaction latency under standard and dialectal speech conditions.

Critical strengths
  1. The paper addresses a relevant evaluation problem by combining automotive tool-use scenarios with full-duplex spoken interaction, a setting that remains relatively underexplored in existing benchmarks.
  2. The proposed evaluation framework considers both final task outcomes and intermediate execution behaviors through NL, DB, and Action verification, potentially providing a more comprehensive assessment than outcome-only evaluation.
Critical weaknesses
  1. The paper lacks sufficient transparency and reproducibility in benchmark construction and evaluation. Although AI-assisted task generation, automated validation, and human spot-checks are mentioned, key details regarding generation procedures, filtering criteria, annotation protocols, and human review coverage are missing. Furthermore, the paper does not provide publicly accessible benchmark data, task specifications, or evaluation code and framework, making it difficult to reproduce the reported results, independently verify benchmark quality, or adopt the benchmark for future research.
  2. The benchmark's representativeness of real-world in-car voice interactions remains insufficiently validated. Tasks are primarily adapted or generated from existing benchmark scenarios, while user speech is simulated through LLMs and TTS with artificially injected noise, without validation against real driving conditions. Moreover, only 25 of the 300 tasks explicitly evaluate turn-management behaviors, all governed by deterministic scripts. This raises concerns about whether the benchmark adequately captures the diversity and unpredictability of real-world full-duplex interactions.
  3. The experimental evaluation focuses primarily on comparing existing voice models, with insufficient validation of the proposed benchmark construction and evaluation design itself. Although the authors introduce task-generation, filtering, action-dependency verification, and multi-dimensional scoring mechanisms, their effectiveness and reliability are not systematically examined. Consequently, the reported model performance differences provide limited evidence of the quality, necessity, or additional evaluation value of these benchmark components.
Questions for authors
  1. Why is natural-language evaluation disabled for the 20 intent-replacement tasks, given that handling user interruptions and goal modifications is a central objective of the benchmark? Beyond verifying updated tool actions and final database states, how does the evaluation protocol assess whether the assistant appropriately acknowledges and responds to interruptions at the conversational level? Could the absence of NL evaluation overlook important failures in full-duplex interaction?
  2. What empirical evidence supports the effectiveness of the proposed benchmark construction and evaluation components? In particular, how do the authors establish that the task-generation and filtering procedures improve task quality, that the redesigned action dependencies capture meaningful execution violations beyond existing evaluation mechanisms, and that the NL and Action validators produce reliable judgments? Since the current experiments predominantly report model-level performance, it remains unclear how the reliability and added value of these benchmark-specific designs are established.
1 comment

We thank the reviewer for the constructive assessment. We address the weaknesses and questions below.

W1 — Transparency and reproducibility. The revision will add an appendix documenting the generation prompts and structured inputs, filtering criteria and thresholds (specification checks, replay, end-to-end trials), the annotation protocol, and human-review coverage (number of tasks, reviewers, agreement). We will release an anonymized package with the 300 task specifications, 64-tool environment, full-duplex harness, evaluation scripts, and configs at https://anonymous.4open.science/r/car-voice-benchmark-8C19/, so the reported results can be reproduced without proprietary keys.

W2 — Representativeness. Only 25 of the 300 tasks are scripted (20 intent-replacement, 5 backchannel); the other 275 use the autonomous interruption/backchannel policy, and the 25 scripted tasks are a controlled stress test rather than the whole benchmark. Tasks are localized Chinese in-car scenarios with redesigned action schemas and explicit dependencies, not a relabeling of CAR-bench. Simulated speech with accent presets and calibrated noise provides controlled acoustic perturbation but not real cabin recordings; we will state this limitation and report a small real-audio pilot. We will also report autonomous barge-in/backchannel behavior beyond the 25 scripted tasks.

W3 — Validation of the benchmark design. The revision adds a validation subsection: (i) all 300 tasks pass schema checks, replay, and end-to-end execution, with generation/filter counts and correction examples; (ii) action-dependency checks catch violations invisible to final states—e.g., a Boolean passed as “True” leaves DB and NL passing but Action at 0; (iii) the NL score is an assertion-level 2-of-3 majority, with 22.6% assertion disagreement and Fleiss’ κ = 0.349 on 3,452 runs; (iv) NL/Action/dependency ablations plus a human audit of validator verdicts.

Q1 — NL scope. The five backchannel tasks require reporting, so NL checks whether the agent continues the report with complete, appropriate content. The 20 intent-replacement tasks target executing the revised goal; their success criteria are DB state and execution trace (required operations and constraints), so we do not add an NL gate on verbal acknowledgment—this reflects operational focus, not indifference to response quality. Interruption responsiveness is distinct from NL content: automated checks did not reliably capture it, so we relied on human listening. We agree this leaves some conversational failures unmeasured; NL is enabled where tasks have explicit spoken requirements, and we acknowledge conversational appropriateness as a complementary dimension.

Q2 — Evidence for components. Trial evaluations across models prioritize manual spot-checks; tasks failed by most models are examined for specification problems. Golden actions are checked for executability, sufficiency, tool/constraint consistency, and absence of redundant operations, with revisions and Appendix B examples. Action evaluation adds signal: on turn-management tasks, gpt-realtime-2.1 and Seeduplex both reach 96.00% DB but 80.00% vs 20.00% Action. For NL, we audit DB/Action-pass but NL-fail cases and 2–1 splits against recorded evidence; Table 4 reports the final NL used in Reward. We acknowledge that manual review is not a comprehensive reliability measurement; systematic independent validation remains future work.

Reviewer_yADLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAR-Voice-Bench combines Chinese automotive tool-use tasks with a full-duplex speech simulation harness and a replayable vehicle environment. Its 300 tasks cover safety, turn management, unavailable tools, disambiguation, and simple or complex workflows. The paper checked success against applicable natural-language requirements, final database states, and action traces, including prerequisites and intent changes. Three streaming speech models are evaluated in noisy standard and dialectal conditions, with three runs each. GPT-Realtime-2.1 has the highest reported overall pass@1. The benchmark also exposes cases where the final database state is correct but the procedure or dialogue is not, and reports tool accuracy and latency diagnostics.

Critical strengths

I like that the benchmark checks the execution trace, not only the final DB state. For example, Seeduplex gets 92.67% DB success but only 15.33% Action success on turn-management tasks (Table 4). That is a meaningful failure mode for tool-using agents that the research community should be aware off.

The 64-tool environment includes explicit causal prerequisites, controlled audio perturbations, and repeated runs. The Chinese in-car scenarios add useful application coverage.

Critical weaknesses

The actual FD evaluation is narrower than the framing suggests - or not putting enough emphasize on being full-duplex - because a turn-taking model can just do as good. Basically, only 25/300 tasks explicitly test turn management: 20 scripted intent replacements and five backchannel cases. Autonomous user interruptions are disabled, and the controller can directly arbitrate assistant audio. Most results therefore measure speech-mediated tool use, not robust floor control. Without an unscripted or matched turn-based control, it is hard to isolate what the full-duplex setting contributes. This is a major weakness that mismatch the broad claim.

The headline model and dialect comparisons use different task populations. Qwen retains only 259 standard and 266 dialectal tasks after unresolved infrastructure errors vs 300 for the other systems (Section 4.1). In the unavailable-tool category it retains only 31/60 and 32/60 tasks (Table 4). The reported 0.94-point dialect drop for Qwen therefore mixes acoustic effects with selection effects. A common-task comparison is needed before interpreting robustness or model rankings.

Questions for authors
  1. What are the scores and paired dialect changes on tasks completed by all three models in both conditions? How do results change if unresolved runs count as failures?

  2. Can the 25 turn-management tasks be evaluated with less scripted turn-taking, or against a text/turn-based control, to isolate the effect of full-duplex behavior?

  3. How were the dialect speech and NL judgments validated?

  4. Should add GPT-Live too.

1 comment

We thank the reviewer for the detailed assessment. Several premises refer to the original submission; the revision changes the evaluation substantially. We address each point below and commit to the additional controls requested.

W1 — Is the full-duplex evaluation narrower than claimed? The revision no longer disables autonomous interruptions: only 25 of the 300 tasks are scripted (20 intent-replacement + 5 backchannel), while the other 275 use the autonomous policy, in which a dedicated decision model decides at each opportunity whether to interrupt or backchannel while the assistant speaks; the 25 scripted tasks serve only as a controlled subset. The On/Off comparison (Table 3) shows that Reward changes in six of ten model–accent cells, with On-average drops of 4.67 points for gpt-realtime-2.1 and 4.83 for gpt-live-1, indicating that floor control is a measurable factor. The orchestrator schedules 200 ms ticks but does not gate assistant audio on turn boundaries; audio streams continuously and interruptions are driven by the user policy. To further isolate the contribution of full duplex, we will add (i) an unscripted variant of the 25 tasks and (ii) a matched text/turn-based control on the same tasks.

W2 — Different task populations and the Qwen comparison. On the common 300 tasks, Qwen’s standard-minus-dialect difference is 3.67 points under On (33.67→30.00) and +1.67 under Off (32.67→34.33; dialect slightly higher).

Q1 — In the latest revision, all five models share the same 300 tasks under both accent conditions. The On-condition standard-minus-dialect gaps are 14.00 (gpt-realtime-2.1), 15.00 (gpt-live-1), 1.33 (Seeduplex), 3.67 (qwen-audio-3.0), and 13.34 (gemini-3.8-live) points; the Off-condition gaps are 17.33, 19.33, −2.33, −1.67, and 14.33. Paired task-level tests and confidence intervals will be added. Counting unresolved runs as failures in Reward would affect evaluation accuracy; in the latest analysis, we obtained complete rewards for all 300 tasks for every model.

Q2 — To better isolate the effect of full-duplex behavior, we compare the remaining 275 non-scripted tasks with autonomous interruptions/backchannels On vs Off; the 25 scripted tasks are kept identical across conditions. On-average Reward changes are: gpt-realtime-2.1 51.00 vs 55.67 (−4.67), gpt-live-1 51.50 vs 56.33 (−4.83), Seeduplex 19.67 vs 20.83 (−1.17), qwen-audio-3.0 31.83 vs 33.50 (−1.67), and gemini-3.8-live 44.00 vs 43.17 (+0.83) points; six of ten model–accent cells decline. This shows that autonomous interruptions/backchannels materially affect task completion on non-scripted tasks.

Q3 — Dialect is applied only on the user side: TTS accent presets (Sichuan and Northeastern, 159/141 tasks) change the user’s speech, not the agent’s. The dialect conditions therefore measure whether the agent can complete the same in-car tasks when the user speaks with an accent, rather than evaluating the agent’s own dialectal speech or comprehension; they primarily reflect task-completion ability under accented input. NL judgments use an assertion-level 2-of-3 majority across GPT-5.2, Kimi-2.5, and DeepSeek-V4.1-Flash.

Q4 — GPT-Live has already been added: the revision evaluates five models, including gpt-live-1 with a gpt-6-astra Responses backend (highest On-average Reward point estimate, 51.50%) and gemini-3.8-live, under four configurations (On/Off × standard/dialect), totaling 6,000 accepted task outcomes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.