- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose to include uncertainty in the reward of GRPO training of speech LLM agent for tool-calling. The uncertainty is estimated via lightweight probes on hidden states of the audio encoder and the LLM decoder, which separates the acoustic tool-routing errors and conditional structured-call failures. The paper also introduces a new benchmark for speech-based tool-calling evaluation.
Critical strengths
- The motivation of including uncertainty signal during training is well-established by the preliminary study in Section 3, making the core idea of the paper clear for the readers.
Critical weaknesses
- As what the paper proposes is a training methodology, it is insufficient to demonstrate the general applicability of the proposed method if the experiment is only conducted on one base model (Qwen3-Omni-30B). Especially, it is unclear whether the observed patterns in Section 3, which motivates the entire study, is a general, common pattern or model-specific. The paper will be more complete if the proposed method is shown to be effective for other speech LLMs.
- The uncertainty is primarily estimated through linear probing in this paper. While Appendix D.4 provides a comparison between such estimation and output-level uncertainty signal, the scope is confined to the failure types instead of the effects of the resulting models if these alternatives were adopted in the proposed training paradigm. Thus, whether these uncertainty signals can still benefit the training process is unclear based on the current results. I will suggest the author to expand the ablation study in Section 5.4 to compare the advantages/disadvantages of using these different signals, which will make the ablation much more comprehensive.
- The author should provide results of some statistical tests to ensure the gap is statistically significant.
- The discussion of related work is shallow. There are several works on speech agent that can also perform tool-calling or evaluating such agents [1,2], yet these works are not discussed in the current manuscript. Discussing these works will make the paper more comprehensive.
[1] Chiang et al., "SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models", ACL 2026
[2] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026
Questions for authors
I have listed some weakness. Please refer to the weakness part. The main text of the paper is not quite self-contained, the reader are asked to refer to the appendix for some important information such as metric definitions, which is a minor comment for the paper readability.
I will also suggest the author to clarify the tool list used during training and evaluation.
4 comments
Response to Reviewer qaLh (Part 2/2)
Weakness 3. Provide statistical tests for the performance gap.
Original Table 5 (now Table 7) reports three-run stability. Section 5.5 and Appendix E.3 now provide the paired test using identical training seeds, splits and evaluation settings.
Table 17. Acc Norm across matched seeds. Mean and sample SD use unrounded values.
| Matched seed | GRPO | UC-GRPO |
|---|---|---|
| 1 | 47.49 | 52.42 |
| 2 | 46.27 | 51.55 |
| 3 | 46.69 | 51.76 |
| Mean ± SD | 46.82 ± 0.62 | 51.91 ± 0.45 |
All three paired gains are positive. A two-sided paired -test gives , , mean gain 5.09 points and 95% CI [4.66, 5.52]. This parametric test describes training-run variation on the fixed evaluation set.
Weakness 4. Discuss SHANKS and Full-Duplex-Bench-v3.
Section 2 now discusses SHANKS [1], which studies simultaneous listening, reasoning and tool use, and Full-Duplex-Bench-v3 [2], which evaluates tool use under natural disfluency. Our contribution concerns internal failure signals guiding policy updates and selective execution. The matched training controls and second-backbone replication test that mechanism under controlled acoustic degradation. The evidence establishes single-turn reliability on the evaluated models.
Question 1. Make important metric and method definitions self-contained in the main text.
Sections 4.1 to 4.4 now state the exact prompt-token feature, risk scores, correctness targets, full reward and inference sequence. Section 5.1 defines the core metrics directly. F1 is the unweighted execute/clarify/direct average before routing, retaining invalid predictions as false negatives. It is computed per acoustic condition and averaged across eleven conditions within each run. Acc Norm checks complete final task success. Tool Hall. pools hallucinated final call-attempt requests over all final call-attempt requests. AST/Param use reference-execute requests, scoring non-execution zero.
Appendix C.1 gives the fixed missing-information and answer-field matching rules, with unresolved responses retained as unsuccessful and no LLM judge. Section 5.2.3 defines When2Call F1 separately: it averages the three reference categories on gated raw choices, retaining direct predictions as false negatives. Acc Norm uses gated length-normalized choices, and Hall. uses the no-tool cannot_answer subset.
Question 2. Clarify the tool lists used for training and evaluation.
Section 5.2.1 and Appendix E.4 now specify identical per-request function names and typed schemas for all compared models. Acoustic variants preserve these inputs. When2Call uses its supplied tools field.
Table 18. Original-request tool inventory. Name groups contain the same function name with different schemas.
| Split | Original requests | Supplied tool IDs | Same-name / different-schema groups | Reference-used tool IDs |
|---|---|---|---|---|
| Train | 8,090 | 1,420 | 37 | 1,180 |
| Validation | 500 | 310 | 6 | 245 |
| Test | 500 | 305 | 5 | 238 |
Table 19. Supplied-interface overlap with training.
| Split | Tool IDs | Seen in Train | Unseen in Train |
|---|---|---|---|
| Validation | 310 | 248 | 62 |
| Test | 305 | 241 | 64 |
Tool identity combines function name and canonical parameter schema, preserving namespaces, required fields, types and constraints. Descriptive annotations/defaults are excluded from the structural hash. Counts cover all supplied tools before acoustic augmentation, with each request counted once per interface. Of the 64 test interfaces unseen in training, 21 occur only in validation and 43 in neither split. These describe tool exposure in a request-held-out benchmark.
[1] Chiang et al., "SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models", ACL 2026.
[2] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026.
Thanks for the response. While the authors provide additional results on Qwen2.5-Omni, this model belongs to the same family as Qwen3-Omni, which is used in the manuscript. Therefore, these results only partially address my concerns regarding cross-model generalization. I encourage the authors to further evaluate models from substantially different families.
At this stage, I do not think the paper meets the bar for a score of 3. Therefore, I will maintain my original overall assessment.
Thank you for clarifying the cross-family generalization concern. We have additionally evaluated Ultravox with a Llama-3.1-8B backbone, extending the experiments beyond Qwen-Omni. This evaluation tests both the training benefit and the representation pattern that motivates our method.
The Ultravox family is also evaluated in Full-Duplex-Bench-v3 [1], which you highlighted as relevant work on spoken tool use. This provides an established benchmark context for our model choice.
With inference routing disabled, learned internal risk improves Acc Norm from 32.62 to 35.45, AST from 0.524 to 0.618, and Param from 0.431 to 0.504 over GRPO. It also exceeds constant, shuffled and entropy risk by 2.16, 2.43 and 1.38 accuracy points, respectively.
Training comparison on Ultravox with Llama-3.1-8B. Routing is disabled throughout.
| Training signal | Acc Norm ↑ | AST ↑ | Param ↑ |
|---|---|---|---|
| Standard GRPO | 32.62 | 0.524 | 0.431 |
| Constant risk | 33.29 | 0.561 | 0.456 |
| Shuffled risk | 33.02 | 0.543 | 0.452 |
| Token entropy | 34.07 | 0.571 | 0.477 |
| Learned internal risk | 35.45 | 0.618 | 0.504 |
The diagnostic also reproduces the complementary ordering observed on both Qwen-Omni backbones. Audio Encoder states lead Decoder states by 0.165 AUROC for routing failure, while Decoder states lead by 0.204 for conditional-call failure. Thus, the non-Qwen experiment supports both the diagnostic finding and its usefulness for policy learning.
Representation diagnostic on the same backbone. Conditional-call failure is evaluated on candidates with the correct tool name.
| Representation | Routing failure AUROC ↑ | Conditional-call failure AUROC ↑ |
|---|---|---|
| Audio Encoder | 0.862 | 0.642 |
| Decoder | 0.697 | 0.846 |
The practical choice of backbones for this validation is narrowed by the need for open weights, direct speech input, accessible Encoder/Decoder states and feasible post-training cost. Within our available resources, we prioritized this additional non-Qwen backbone. Extending the study to Phi-4-multimodal-instruct and GLM-4-Voice is a natural next step. The current results establish replication across Qwen-Omni and a Llama-based speech model, while broader coverage remains future work.
[1] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026.
Response to Reviewer qaLh (Part 1/2)
Thanks for your careful and valuable comments. These comments have been of great value to my approach. We will explain your concerns point by point.
Section 3 identifies complementary predictive signals for routing and conditional-call failure. To test their applicability beyond Qwen3-Omni-30B, Section 5.5 now reports both policy training and the matched-target diagnostic on Qwen2.5-Omni-7B.
Table 5(a). Complete second-backbone training comparison. Routing is off throughout.
Table 5(b). Matched-target AUROC. Conditional failure uses correct-tool candidates. Best values are marked within each backbone.
Learned risk improves 7B accuracy by 3.31 points over GRPO, with higher AST/Param. Audio leads Decoder by 0.191 AUROC for routing failure, while Decoder leads by 0.227 for conditional-call failure. Both states use the same held-out examples per target. Probes use training-only fitting and validation selection. This replicates both the motivating pattern and its training utility on the second Qwen-Omni backbone.
The original Appendix D.4 compares failure prediction. Section 5.4 now also tests token entropy inside policy training, with constant and shuffled risk as controls.
Table 4. Matched training objectives. F1 scores candidates before routing. On/Off reuse a checkpoint and candidates.
The four risk-conditioned variants share bonuses, penalties, coefficients and loss weighting. Constant risk uses mean training risk, and shuffled risk permutes scores within each acoustic condition. Initialization, data, rollout budget and optimizer are matched.
With routing off, entropy improves accuracy from 46.82 to 49.02. Learned risk further reaches 50.42, exceeding entropy by 1.40 accuracy points, 0.070 AST and 0.057 Param. AST/Param use the fixed reference-execute population with non-execution scored zero. The internal signal is available from prompt states before generation. Self-consistency and ASR confidence remain diagnostic comparisons in Table 10.
Table 4's F1=66.54 and Table 1's three-run mean 66.50 use the same definition and condition averaging.