- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CAR-Voice-Bench, a benchmark for evaluating full-duplex voice assistants in automotive scenarios. Building upon CAR-bench and τ-Voice, the authors construct 300 Chinese in-car tasks across six categories and develop an evaluation framework combining natural-language (NL), database-state (DB), and action-trace verification. Experiments on three existing voice models examine task completion, tool-use accuracy, and interaction latency under standard and dialectal speech conditions.
Critical strengths
- The paper addresses a relevant evaluation problem by combining automotive tool-use scenarios with full-duplex spoken interaction, a setting that remains relatively underexplored in existing benchmarks.
- The proposed evaluation framework considers both final task outcomes and intermediate execution behaviors through NL, DB, and Action verification, potentially providing a more comprehensive assessment than outcome-only evaluation.
Critical weaknesses
- The paper lacks sufficient transparency and reproducibility in benchmark construction and evaluation. Although AI-assisted task generation, automated validation, and human spot-checks are mentioned, key details regarding generation procedures, filtering criteria, annotation protocols, and human review coverage are missing. Furthermore, the paper does not provide publicly accessible benchmark data, task specifications, or evaluation code and framework, making it difficult to reproduce the reported results, independently verify benchmark quality, or adopt the benchmark for future research.
- The benchmark's representativeness of real-world in-car voice interactions remains insufficiently validated. Tasks are primarily adapted or generated from existing benchmark scenarios, while user speech is simulated through LLMs and TTS with artificially injected noise, without validation against real driving conditions. Moreover, only 25 of the 300 tasks explicitly evaluate turn-management behaviors, all governed by deterministic scripts. This raises concerns about whether the benchmark adequately captures the diversity and unpredictability of real-world full-duplex interactions.
- The experimental evaluation focuses primarily on comparing existing voice models, with insufficient validation of the proposed benchmark construction and evaluation design itself. Although the authors introduce task-generation, filtering, action-dependency verification, and multi-dimensional scoring mechanisms, their effectiveness and reliability are not systematically examined. Consequently, the reported model performance differences provide limited evidence of the quality, necessity, or additional evaluation value of these benchmark components.
Questions for authors
- Why is natural-language evaluation disabled for the 20 intent-replacement tasks, given that handling user interruptions and goal modifications is a central objective of the benchmark? Beyond verifying updated tool actions and final database states, how does the evaluation protocol assess whether the assistant appropriately acknowledges and responds to interruptions at the conversational level? Could the absence of NL evaluation overlook important failures in full-duplex interaction?
- What empirical evidence supports the effectiveness of the proposed benchmark construction and evaluation components? In particular, how do the authors establish that the task-generation and filtering procedures improve task quality, that the redesigned action dependencies capture meaningful execution violations beyond existing evaluation mechanisms, and that the NL and Action validators produce reliable judgments? Since the current experiments predominantly report model-level performance, it remains unclear how the reliability and added value of these benchmark-specific designs are established.
We thank the reviewer for the constructive assessment. We address the weaknesses and questions below.
W1 — Transparency and reproducibility. The revision will add an appendix documenting the generation prompts and structured inputs, filtering criteria and thresholds (specification checks, replay, end-to-end trials), the annotation protocol, and human-review coverage (number of tasks, reviewers, agreement). We will release an anonymized package with the 300 task specifications, 64-tool environment, full-duplex harness, evaluation scripts, and configs at https://anonymous.4open.science/r/car-voice-benchmark-8C19/, so the reported results can be reproduced without proprietary keys.
W2 — Representativeness. Only 25 of the 300 tasks are scripted (20 intent-replacement, 5 backchannel); the other 275 use the autonomous interruption/backchannel policy, and the 25 scripted tasks are a controlled stress test rather than the whole benchmark. Tasks are localized Chinese in-car scenarios with redesigned action schemas and explicit dependencies, not a relabeling of CAR-bench. Simulated speech with accent presets and calibrated noise provides controlled acoustic perturbation but not real cabin recordings; we will state this limitation and report a small real-audio pilot. We will also report autonomous barge-in/backchannel behavior beyond the 25 scripted tasks.
W3 — Validation of the benchmark design. The revision adds a validation subsection: (i) all 300 tasks pass schema checks, replay, and end-to-end execution, with generation/filter counts and correction examples; (ii) action-dependency checks catch violations invisible to final states—e.g., a Boolean passed as “True” leaves DB and NL passing but Action at 0; (iii) the NL score is an assertion-level 2-of-3 majority, with 22.6% assertion disagreement and Fleiss’ κ = 0.349 on 3,452 runs; (iv) NL/Action/dependency ablations plus a human audit of validator verdicts.
Q1 — NL scope. The five backchannel tasks require reporting, so NL checks whether the agent continues the report with complete, appropriate content. The 20 intent-replacement tasks target executing the revised goal; their success criteria are DB state and execution trace (required operations and constraints), so we do not add an NL gate on verbal acknowledgment—this reflects operational focus, not indifference to response quality. Interruption responsiveness is distinct from NL content: automated checks did not reliably capture it, so we relied on human listening. We agree this leaves some conversational failures unmeasured; NL is enabled where tasks have explicit spoken requirements, and we acknowledge conversational appropriateness as a complementary dimension.
Q2 — Evidence for components. Trial evaluations across models prioritize manual spot-checks; tasks failed by most models are examined for specification problems. Golden actions are checked for executability, sufficiency, tool/constraint consistency, and absence of redundant operations, with revisions and Appendix B examples. Action evaluation adds signal: on turn-management tasks, gpt-realtime-2.1 and Seeduplex both reach 96.00% DB but 80.00% vs 20.00% Action. For NL, we audit DB/Action-pass but NL-fail cases and 2–1 splits against recorded evidence; Table 4 reports the final NL used in Reward. We acknowledge that manual review is not a comprehensive reliability measurement; systematic independent validation remains future work.