acceptodds
Under review as a conference paper at ICLR 2027

From Speech to Action: Uncertainty-Aware Reinforcement Learning for Tool Calling in Speech LLMs

Abstract

Voice-driven tool calling extends LLM agents from typed interaction to spoken requests, enabling users to invoke external services through natural speech. However, unlike text-based tool calling, speech input introduces an upstream acoustic stage where noise, speaker variation, and ambiguity can distort the request before tool-use decisions are made. Existing uncertainty-aware tool-use methods mainly target text agents, leaving speech-channel uncertainty in tool routing and structured-call generation underexplored. To address this gap, we introduce a hierarchical diagnostic view that separates observable tool-name routing failures from conditional structured-call failures after correct routing, showing that these failure risks can be predicted from Speech LLM representations. This motivates UC-GRPO, an uncertainty-aware post-training and inference framework for end-to-end Speech LLM tool calling. UC-GRPO extracts routing and conditional decisional uncertainty with lightweight probes over Audio Encoder and Decoder states, uses the probe-derived signals to shape GRPO rewards, and implements selective execution through clarification routing. To enable systematic evaluation under controlled acoustic degradation, we further propose SpeechToolBench, a noise-augmented benchmark combining public tool-calling tasks with practical voice-agent scenarios and routing-decision diagnostics. UC-GRPO achieves 51.91% Acc Norm on SpeechToolBench, improving over the base Speech LLM by 11.46 points and standard GRPO by 5.09 points. Zero-shot evaluation on audio-adapted When2Call further shows that the learned uncertainty-aware behavior transfers to an independent tool-use benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech LLMs, Tool Calling, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qaLhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose to include uncertainty in the reward of GRPO training of speech LLM agent for tool-calling. The uncertainty is estimated via lightweight probes on hidden states of the audio encoder and the LLM decoder, which separates the acoustic tool-routing errors and conditional structured-call failures. The paper also introduces a new benchmark for speech-based tool-calling evaluation.

Critical strengths
  1. The motivation of including uncertainty signal during training is well-established by the preliminary study in Section 3, making the core idea of the paper clear for the readers.
Critical weaknesses
  1. As what the paper proposes is a training methodology, it is insufficient to demonstrate the general applicability of the proposed method if the experiment is only conducted on one base model (Qwen3-Omni-30B). Especially, it is unclear whether the observed patterns in Section 3, which motivates the entire study, is a general, common pattern or model-specific. The paper will be more complete if the proposed method is shown to be effective for other speech LLMs.
  2. The uncertainty is primarily estimated through linear probing in this paper. While Appendix D.4 provides a comparison between such estimation and output-level uncertainty signal, the scope is confined to the failure types instead of the effects of the resulting models if these alternatives were adopted in the proposed training paradigm. Thus, whether these uncertainty signals can still benefit the training process is unclear based on the current results. I will suggest the author to expand the ablation study in Section 5.4 to compare the advantages/disadvantages of using these different signals, which will make the ablation much more comprehensive.
  3. The author should provide results of some statistical tests to ensure the gap is statistically significant.
  4. The discussion of related work is shallow. There are several works on speech agent that can also perform tool-calling or evaluating such agents [1,2], yet these works are not discussed in the current manuscript. Discussing these works will make the paper more comprehensive.

[1] Chiang et al., "SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models", ACL 2026

[2] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026

Questions for authors

I have listed some weakness. Please refer to the weakness part. The main text of the paper is not quite self-contained, the reader are asked to refer to the appendix for some important information such as metric definitions, which is a minor comment for the paper readability.

I will also suggest the author to clarify the tool list used during training and evaluation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qaLh (Part 1/2)

Thanks for your careful and valuable comments. These comments have been of great value to my approach. We will explain your concerns point by point.

Weakness 1. General applicability beyond Qwen3-Omni-30B and whether the Section 3 pattern is model-specific.

Section 3 identifies complementary predictive signals for routing and conditional-call failure. To test their applicability beyond Qwen3-Omni-30B, Section 5.5 now reports both policy training and the matched-target diagnostic on Qwen2.5-Omni-7B.

Table 5(a). Complete second-backbone training comparison. Routing is off throughout.

Training signalAcc Norm ↑AST ↑Param ↑
Standard GRPO31.820.5100.423
Constant risk33.020.5490.445
Shuffled risk32.730.5330.438
Token entropy33.470.5550.457
Learned internal risk35.130.5920.486

Table 5(b). Matched-target AUROC. Conditional failure uses correct-tool candidates. Best values are marked within each backbone.

BackboneRepresentationRouting failure AUROC ↑Conditional-call failure AUROC ↑
Qwen3-Omni-30BAudio Encoder0.9200.658
Qwen3-Omni-30BDecoder0.7060.912
Qwen2.5-Omni-7BAudio Encoder0.8740.631
Qwen2.5-Omni-7BDecoder0.6830.858

Learned risk improves 7B accuracy by 3.31 points over GRPO, with higher AST/Param. Audio leads Decoder by 0.191 AUROC for routing failure, while Decoder leads by 0.227 for conditional-call failure. Both states use the same held-out examples per target. Probes use training-only fitting and validation selection. This replicates both the motivating pattern and its training utility on the second Qwen-Omni backbone.

Weakness 2. Compare alternative uncertainty signals within policy training, rather than only diagnostic AUROC.

The original Appendix D.4 compares failure prediction. Section 5.4 now also tests token entropy inside policy training, with constant and shuffled risk as controls.

Table 4. Matched training objectives. F1 scores candidates before routing. On/Off reuse a checkpoint and candidates.

Training signalRoutingAcc Norm ↑F1 ↑Tool Hall. ↓AST ↑Param ↑
Standard GRPOOff46.8263.7852.860.6430.551
Constant riskOff48.3664.5751.380.7160.585
Shuffled riskOff48.1364.2651.810.7030.579
Token entropyOff49.0264.9450.660.7420.613
Learned internal riskOff50.4266.5449.050.8120.670
Learned internal riskOn51.9166.5447.450.8030.660

The four risk-conditioned variants share bonuses, penalties, coefficients and loss weighting. Constant risk uses mean training risk, and shuffled risk permutes scores within each acoustic condition. Initialization, data, rollout budget and optimizer are matched.

With routing off, entropy improves accuracy from 46.82 to 49.02. Learned risk further reaches 50.42, exceeding entropy by 1.40 accuracy points, 0.070 AST and 0.057 Param. AST/Param use the fixed reference-execute population with non-execution scored zero. The internal signal is available from prompt states before generation. Self-consistency and ASR confidence remain diagnostic comparisons in Table 10.

Table 4's F1=66.54 and Table 1's three-run mean 66.50 use the same definition and condition averaging.

AuthorsOpenReview ↗

Response to Reviewer qaLh (Part 2/2)

Weakness 3. Provide statistical tests for the performance gap.

Original Table 5 (now Table 7) reports three-run stability. Section 5.5 and Appendix E.3 now provide the paired test using identical training seeds, splits and evaluation settings.

Table 17. Acc Norm across matched seeds. Mean and sample SD use unrounded values.

Matched seedGRPOUC-GRPO
147.4952.42
246.2751.55
346.6951.76
Mean ± SD46.82 ± 0.6251.91 ± 0.45

All three paired gains are positive. A two-sided paired tt-test gives t(2)=50.839t(2)=50.839, p=0.0004p=0.0004, mean gain 5.09 points and 95% CI [4.66, 5.52]. This parametric test describes training-run variation on the fixed evaluation set.

Weakness 4. Discuss SHANKS and Full-Duplex-Bench-v3.

Section 2 now discusses SHANKS [1], which studies simultaneous listening, reasoning and tool use, and Full-Duplex-Bench-v3 [2], which evaluates tool use under natural disfluency. Our contribution concerns internal failure signals guiding policy updates and selective execution. The matched training controls and second-backbone replication test that mechanism under controlled acoustic degradation. The evidence establishes single-turn reliability on the evaluated models.

Question 1. Make important metric and method definitions self-contained in the main text.

Sections 4.1 to 4.4 now state the exact prompt-token feature, risk scores, correctness targets, full reward and inference sequence. Section 5.1 defines the core metrics directly. F1 is the unweighted execute/clarify/direct average before routing, retaining invalid predictions as false negatives. It is computed per acoustic condition and averaged across eleven conditions within each run. Acc Norm checks complete final task success. Tool Hall. pools hallucinated final call-attempt requests over all final call-attempt requests. AST/Param use reference-execute requests, scoring non-execution zero.

Appendix C.1 gives the fixed missing-information and answer-field matching rules, with unresolved responses retained as unsuccessful and no LLM judge. Section 5.2.3 defines When2Call F1 separately: it averages the three reference categories on gated raw choices, retaining direct predictions as false negatives. Acc Norm uses gated length-normalized choices, and Hall. uses the no-tool cannot_answer subset.

Question 2. Clarify the tool lists used for training and evaluation.

Section 5.2.1 and Appendix E.4 now specify identical per-request function names and typed schemas for all compared models. Acoustic variants preserve these inputs. When2Call uses its supplied tools field.

Table 18. Original-request tool inventory. Name groups contain the same function name with different schemas.

SplitOriginal requestsSupplied tool IDsSame-name / different-schema groupsReference-used tool IDs
Train8,0901,420371,180
Validation5003106245
Test5003055238

Table 19. Supplied-interface overlap with training.

SplitTool IDsSeen in TrainUnseen in Train
Validation31024862
Test30524164

Tool identity combines function name and canonical parameter schema, preserving namespaces, required fields, types and constraints. Descriptive annotations/defaults are excluded from the structural hash. Counts cover all supplied tools before acoustic augmentation, with each request counted once per interface. Of the 64 test interfaces unseen in training, 21 occur only in validation and 43 in neither split. These describe tool exposure in a request-held-out benchmark.

[1] Chiang et al., "SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models", ACL 2026.

[2] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026.

Thanks for the response. While the authors provide additional results on Qwen2.5-Omni, this model belongs to the same family as Qwen3-Omni, which is used in the manuscript. Therefore, these results only partially address my concerns regarding cross-model generalization. I encourage the authors to further evaluate models from substantially different families.

At this stage, I do not think the paper meets the bar for a score of 3. Therefore, I will maintain my original overall assessment.

Thank you for clarifying the cross-family generalization concern. We have additionally evaluated Ultravox with a Llama-3.1-8B backbone, extending the experiments beyond Qwen-Omni. This evaluation tests both the training benefit and the representation pattern that motivates our method.

The Ultravox family is also evaluated in Full-Duplex-Bench-v3 [1], which you highlighted as relevant work on spoken tool use. This provides an established benchmark context for our model choice.

With inference routing disabled, learned internal risk improves Acc Norm from 32.62 to 35.45, AST from 0.524 to 0.618, and Param from 0.431 to 0.504 over GRPO. It also exceeds constant, shuffled and entropy risk by 2.16, 2.43 and 1.38 accuracy points, respectively.

Training comparison on Ultravox with Llama-3.1-8B. Routing is disabled throughout.

Training signalAcc Norm ↑AST ↑Param ↑
Standard GRPO32.620.5240.431
Constant risk33.290.5610.456
Shuffled risk33.020.5430.452
Token entropy34.070.5710.477
Learned internal risk35.450.6180.504

The diagnostic also reproduces the complementary ordering observed on both Qwen-Omni backbones. Audio Encoder states lead Decoder states by 0.165 AUROC for routing failure, while Decoder states lead by 0.204 for conditional-call failure. Thus, the non-Qwen experiment supports both the diagnostic finding and its usefulness for policy learning.

Representation diagnostic on the same backbone. Conditional-call failure is evaluated on candidates with the correct tool name.

RepresentationRouting failure AUROC ↑Conditional-call failure AUROC ↑
Audio Encoder0.8620.642
Decoder0.6970.846

The practical choice of backbones for this validation is narrowed by the need for open weights, direct speech input, accessible Encoder/Decoder states and feasible post-training cost. Within our available resources, we prioritized this additional non-Qwen backbone. Extending the study to Phi-4-multimodal-instruct and GLM-4-Voice is a natural next step. The current results establish replication across Qwen-Omni and a Llama-based speech model, while broader coverage remains future work.

[1] Lin et al., "Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency", Preprint, 2026.

Reviewer_PczwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies uncertainty-aware tool calling in end-to-end speech LLM. It separates tool-name routing failures from structured-call failures and train lightweight probes over Audio Encoder and Decoder representations to predict these risks. The proposed UC-GRPO framework uses the resulting uncertainty estimates for reward shaping, loss weighting and inference time routing to clarification. The paper also introduces Speech Tool Bench, combining public tool-calling tasks and human-recorded requests under controlled acoustic perturbations. UC-GRPO improves normalized accuracy from 46.82% to 51.91% over standard GRPO, with smaller gains on audio-adapted When2Call.

Critical strengths
  1. Useful diagnostic analysis. Tab 7 compares both representations on the same targets. Audio Encoder and Decoder states best predict routing and conditional structured-call failures, respectively, with AUROC of 0.920 and 0.912. These results support complementary failure signals without implying a causal separation.
  2. Consistent gains beyond abstention. UC-GRPO improve accuracy over standard GRPO by 5.09 percentage points with consistent results across three seeds. AST and parameter accuracy also improve despite abstentions receiving zero on reference-execute examples. The routing-only ablation and zero-shot When2Cal results provide additional support.
Critical weaknesses
  1. Uncertainty-specific gains are not isolated. UC-GRPO changes execution penalties, clarification bonuses, and loss weighting alongside uncertainty estimation. Existing ablations remove these incentives rather than control for them so improvements may reflect cost-sensitive optimization. A reward-matched baseline using constant or shufled uncertainty would help establish the contribution of learned risk estimates
  2. Clarification supervision and evaluation are underspecified. The construction and distribution of clarification and direct-answer labels are unclear. Unchanged reference decisions across acoustic variants raise questions about how noise-induced ambiguity is scored. The task reward and Fig 4’s combination of reference based scoring with routing also need clarification, since these details affect interpretation of the action-selection gains.
Questions for authors
  1. Can you please compare UC-GRPO against a reward-matched control using constant or shuffled uncertainty, while retaining the same clarification bonus, erroneous-execution penalty, and loss-weighting structure?
  2. How are execute, clarify, and direct-answer targets constructed and annotated, and what are their counts in each split?
  3. Can you please specify the complete task reward, the exact decoder token position used for the action-decision state, and how uncertainty is computed for clarification rollouts? Also in fig 4does inference-time routing operates without access to reference decisions?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Pczw (Part 1/2)

Thanks for your careful and valuable comments. These comments have been of great value to my approach. We will explain your concerns point by point.

Weakness 1. Uncertainty-specific gains are not isolated.

Table 1 establishes a 5.09-point gain over GRPO with higher AST/Param, and Table 3 shows a benefit beyond routing alone. The reward-matched controls under Question 1 isolate informative risk within that objective.

Weakness 2. Clarification supervision and evaluation are underspecified.

Appendix B and C.1 now make the target construction and content scoring explicit. Labels precede acoustic augmentation, so the evaluation tests the same intended task across speech conditions. The counts and noise-scoring rule are given under Question 2.

Question 1. Compare against constant or shuffled uncertainty while retaining the clarification bonus, execution penalty and loss weighting.

Section 5.4 now includes constant and shuffled risk, retaining the clarification bonus, erroneous-execution penalty, coefficients and loss-weighting structure. Constant risk uses mean training risk, while shuffled risk permutes scores within each acoustic condition. Initialization, data, rollout budget and optimizer are matched.

Table 4. Matched training objectives. F1 scores candidates before routing. On/Off reuse a checkpoint and candidates.

Training signalRoutingAcc Norm ↑F1 ↑Tool Hall. ↓AST ↑Param ↑
Standard GRPOOff46.8263.7852.860.6430.551
Constant riskOff48.3664.5751.380.7160.585
Shuffled riskOff48.1364.2651.810.7030.579
Token entropyOff49.0264.9450.660.7420.613
Learned internal riskOff50.4266.5449.050.8120.670
Learned internal riskOn51.9166.5447.450.8030.660

With routing off, learned risk exceeds constant/shuffled/entropy by 2.06/2.29/1.40 accuracy points. Higher AST/Param on the fixed reference-execute population, with non-execution scored zero, support improved call generation. Routing then adds 1.49 points on the same checkpoint. The 66.54 F1 here and Table 1's three-run mean 66.50 use the same definition and condition averaging.

Question 2. How are execute, clarify and direct targets constructed, and what are their counts?

Execute targets provide enough information for a supported call. Clarify targets remove required information or introduce tool-critical ambiguity and record what must be asked. Direct targets specify the expected no-tool answer or inability requirement (Appendix B).

Reference-action counts before acoustic augmentation (Section 5.1).

SplitExecuteClarifyDirectTotal
Train7,600400908,090
Validation470255500
Test480155500
Total8,5504401009,090

The test set becomes 5,280 execute, 165 clarify and 55 direct examples across eleven conditions. References are preserved to measure recovery of the intended task under degradation. Noise does not automatically relabel execute as clarify. Clarification on a reference-execute request may be an operational fallback, but receives no single-turn task-success, AST or parameter credit.

Clarification must ask for the reference-required missing information and only permitted targets. Structured targets are checked first, with fixed question/alias fallback only when those targets are absent. An explicit wrong target, guessed missing value or tool invocation is incorrect. Direct success requires no tool invocation, the expected subtype and correct required field values. These fixed predicates score both rewards and evaluation. No LLM judge is used. Unresolved responses remain in the full accuracy denominator as unsuccessful (Appendix C.1).

AuthorsOpenReview ↗

Response to Reviewer Pczw (Part 2/2)

Question 3. Specify the complete reward, exact Decoder token, uncertainty for clarification, and whether inference routing accesses references.

Sections 4.1 to 4.4 now state the requested features, scores and reward. Figure 4 separates reference-free online routing from offline scoring.

  1. Complete reward.

Both GRPO and UC-GRPO use

Rtask=0.10F+I[ai=execute](0.30N+0.60P)+D.R_{\mathrm{task}}=0.10F+\mathbb I[a_i=\mathrm{execute}](0.30N+0.60P)+D.

FF checks format, NN the exact tool-name multiset, PP normalized parameter accuracy, and DD task-correct non-execute content. N=P=0N=P=0 without a reference call, and unparseable outputs receive zero task reward. The UC-GRPO terms are

R_{\mathrm{uc},i}&=R_{\mathrm{task},i} +4.0U_i\mathbb I[a_i=\mathrm{clarify}]C_{\mathrm{clar}}(z_i)\\ &\quad-2.5U_i\mathbb I[a_i=\mathrm{execute}]\mathbb I[C_{\mathrm{call}}(z_i)=0],\\ W_i&=1+U_i\mathbb I[a_i=\mathrm{execute}] \mathbb I[C_{\mathrm{call}}(z_i)=0]\mathbb I[\hat A_i<0]. \end{aligned}$$ Here $U_i=U_{\mathrm{full}}(x,\mathcal T)$, $C_{\mathrm{clar}}$ checks the reference missing-information target, and $C_{\mathrm{call}}$ checks the complete normalized call at $\delta=1.0$. Execution on a non-execute reference has $C_{\mathrm{call}}=0$. Unresolved non-execute responses have $D=C_{\mathrm{clar}}=0$. $\hat A_i$ is the group-normalized advantage from $R_{\mathrm{uc}}$. The coefficients are $\alpha=4.0$, $\beta=2.5$, $\lambda=1.0$. $W_i$ multiplies only the policy/GRPO loss, leaving the original KL coefficient unchanged. 2. Exact token and risk. The Decoder feature is the final-layer state at the last non-padding prompt token, before the first generated response token. The Audio feature is the temporal mean of the deepest Encoder layer. The scores are $$\begin{aligned} U_{\mathrm{route}}&=\sigma(w_r^\top h^{\mathrm{aud}}+b_r),\\ U_{\mathrm{dec}}&=\sigma(w_d^\top h^{\mathrm{dec}}+b_d),\\ U_{\mathrm{full}}&=\sigma(w_f^\top[U_{\mathrm{route}},U_{\mathrm{dec}}]+b_f). \end{aligned}$$ Probes are fitted on training data with validation selection. Their parameters stay fixed during RL, while detached scores are recomputed from current hidden states. The routing target is tool-name failure. Conditional-call failure is fitted only on correct-tool candidates, masking wrong-tool candidates. The fusion target is complete-call failure. 3. Clarification rollouts. All rollouts of one request share the same prompt-side risk. Clarification text is checked against its reference missing-information targets for reward, but is not a probe feature or a tool-call probe target. 4. Inference access to references. For SpeechToolBench, prefill computes risk from speech and supplied tools. Native clarify/direct candidates are retained. An execute candidate with $U_{\mathrm{full}}>0.5$ is withheld and a second pass generates clarification, otherwise it proceeds to execution. References are used only by the separate offline scorer after the final response is fixed. When2Call instead gates each MCQ likelihood stream by changing high-risk tool_call choices to request_for_info. It selects an existing option and does not generate a second clarification. The threshold transfers without When2Call tuning.

Thanks very much for the thorough response, it addresses my concerns and I've raised my score accordingly.

Thank you for your thoughtful feedback and for updating your assessment. We appreciate your careful consideration of our responses.

Reviewer_dC9cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies reliable tool calling from spoken requests, particularly under acoustic degradation. It distinguishes two observable failure types: selecting the wrong tool and producing an incorrect structured call after selecting the correct tool. Lightweight probes over Audio Encoder and Decoder representations predict these risks, and a fusion head estimates overall execution risk. The proposed UC-GRPO uses this estimate to modify GRPO rewards and loss weights, and to redirect high-risk executable candidates to clarification before execution.

The paper also introduces SpeechToolBench, comprising 9,090 original requests, including 1,000 human-recorded practical scenarios. Its test set contains 500 requests evaluated under eleven acoustic conditions. On Qwen3-Omni-30B, the paper reports normalized accuracy of 51.91%, compared with 46.82% for standard GRPO and 40.45% for the base model.

Critical strengths

The problem and operational framing are well chosen. The matched training comparison is meaningful. The structured-call gains are informative. The appendix addresses several obvious alternative explanations.

Critical weaknesses
  1. The reported metrics are internally inconsistent across multiple places. There is also consistency question concerning macro-F1.

  2. The When2Call adaptation needs an explicit label mapping.

  3. The method is not specified precisely enough to be reproduced. For example, the phrase “pre-execution action-decision state” does not identify the token position. The score and correctness labels are incomplete.

  4. The ablations do not isolate the value of learned uncertainty from changes to action costs.

  5. Benchmark generalization and statistical support need more detail.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dC9c (Part 1/3)

Thanks for your careful and valuable comments. These comments have been of great value to my approach. We will explain your concerns point by point.

Weakness 1. The reported metrics are internally inconsistent across multiple places. There is also consistency question concerning macro-F1.

Section 5.1 and Appendix C.1 now state the common definition explicitly. Every SpeechToolBench F1 result is the unweighted average of execute, clarify and direct F1 on candidates before routing. Invalid predictions remain false negatives of their reference classes. F1 is calculated per acoustic condition, averaged over eleven conditions within each run, then summarized across runs. Table 1 reports 66.50 across three runs. The same-checkpoint On/Off rows in Table 4 have identical F1.

Original Table 10, now Table 12, measures complete task outcomes after routing. A correct execute label can still contain wrong arguments. Its mean correct counts recover Table 1 accuracy: (2,738+117)/5,500×100=51.91%(2,738+117)/5,500\times100=51.91\%. The ratio 117/225=52.00%117/225=52.00\% is pooled non-execute task success, not clarify precision or another F1.

Table 12. Final-response diagnostic. Counts and uncertainty are three-seed means. Correctness includes response content.

Final response / task outcomeMean NMean uncertainty
Execute (all)52750.329
Task-correct execution27380.320
Task-incorrect execution25370.338
Non-execute (all)2250.506
Task-correct non-execute1170.542
Task-incorrect non-execute1080.466

Macro averaging preserves minority-action performance under the 96%/3%/1% test proportions. Task accuracy checks whether the complete request succeeds. When2Call [1] likewise separates behavior selection from call validation. Its benchmark-specific metric is explicitly named When2Call F1 in Table 2.

Weakness 2. The When2Call adaptation needs an explicit label mapping.

Section 5.2.3 and Appendix C.2 now give the mapping used for Table 2. Audio conversion preserves the supplied tools and reference decisions. We score the four answer continuations by conditional likelihood [1].

Candidate categoryPolicy actionScoring
tool_callexecuteCorrect only for a tool_call reference
request_for_infoclarifyCorrect only for a request_for_info reference
directdirectAn incorrect answer candidate
cannot_answerdirectDistinct from direct in scoring

Raw choices maximize candidate log-likelihood, and normalized choices maximize log-likelihood divided by candidate length. Each stream is gated separately: tool_call changes to request_for_info when Ufull>0.5U_{\mathrm{full}}>0.5, while other choices remain unchanged. The threshold transfers without When2Call tuning.

When2Call F1 uses gated raw choices and averages tool_call, request_for_info and cannot_answer F1. Direct predictions remain false negatives for their reference categories, without adding a fourth class to the average. Tool Hall. uses the same raw choices on no-tool cannot_answer requests. Acc Norm uses gated length-normalized choices. There is no semantic judge or second clarification generation.

[1] Ross et al., "When2Call: When (not) to Call Tools", NAACL 2025.

AuthorsOpenReview ↗

Response to Reviewer dC9c (Part 2/3)

Weakness 3. The method is not specified precisely enough to be reproduced. The token position, score and correctness labels are incomplete.

Sections 4.1 to 4.4 and Appendix C.1 now specify the requested token, scores and labels. Figure 4 separates online routing from offline scoring.

  1. Features and scores.

The Decoder feature is the final-layer state at the last non-padding prompt token, before the first response token. The Audio feature is the temporal mean of the deepest Encoder layer. The scores are

U_{\mathrm{route}}&=\sigma(w_r^\top h^{\mathrm{aud}}+b_r),\\ U_{\mathrm{dec}}&=\sigma(w_d^\top h^{\mathrm{dec}}+b_d),\\ U_{\mathrm{full}}&=\sigma(w_f^\top[U_{\mathrm{route}},U_{\mathrm{dec}}]+b_f). \end{aligned}$$ Probes are fitted on training data with settings selected on validation data. Their parameters stay fixed during RL, while hidden states and detached scores are recomputed from the current policy. Rollouts of the same request share this prompt-side risk. 2. Correctness targets. $C_{\mathrm{name}}=1$ requires an exact tool-name-multiset match. $C_{\mathrm{call}}=1$ requires a complete normalized call match at $\delta=1.0$. Probe failure labels are $$\begin{aligned} y_{\mathrm{route}}&=1-C_{\mathrm{name}},\\ y_{\mathrm{dec}}&=1-C_{\mathrm{call}}\quad\text{only on }C_{\mathrm{name}}=1,\\ y_{\mathrm{full}}&=1-C_{\mathrm{call}}. \end{aligned}$$ Wrong-tool candidates are masked out of conditional-probe fitting. Clarify/direct rollouts have their own decision targets and no tool-call probe labels. Exact calls require a supported tool, all required fields, matching scored values including optional values, and no illegal extra fields. jsonschema validates types. Recursive field scores use equal weights after string, numeric, enum and temporal normalization. Multiple calls are grouped by tool name and matched by maximum-weight Hungarian assignment, with unmatched calls scored zero. Clarification must request the reference-required missing information and contain only permitted information requests. Structured targets are checked first. Fixed question patterns and aliases are used only when structured targets are absent and cannot override an explicit wrong target. A parameter mention alone, a guessed missing value or a tool invocation is insufficient. Direct success requires no tool invocation, the expected answer subtype and correct required field values using the fixed type-specific comparators and declared tolerances. Inability responses must satisfy the reference inability requirement. No LLM judge is used. Unresolved responses remain unsuccessful in the full accuracy denominator. 3. Reward and weighting. The task reward shared with GRPO is $$R_{\mathrm{task}}=0.10F+\mathbb I[a_i=\mathrm{execute}](0.30N+0.60P)+D.$$ $F$ is format validity, $N$ exact tool-name-multiset correctness, $P$ parameter accuracy, and $D$ correct non-execute content. Without a reference call, $N=P=0$. Unparseable outputs receive zero task reward. Writing $U_i=U_{\mathrm{full}}(x,\mathcal T)$, the uncertainty terms are $$\begin{aligned} R_{\mathrm{uc},i}&=R_{\mathrm{task},i} +4.0U_i\mathbb I[a_i=\mathrm{clarify}]C_{\mathrm{clar}}(z_i)\\ &\quad-2.5U_i\mathbb I[a_i=\mathrm{execute}]\mathbb I[C_{\mathrm{call}}(z_i)=0],\\ W_i&=1+U_i\mathbb I[a_i=\mathrm{execute}] \mathbb I[C_{\mathrm{call}}(z_i)=0]\mathbb I[\hat A_i<0]. \end{aligned}$$ $C_{\mathrm{clar}}$ uses the same clarification predicate as evaluation. Unresolved non-execute responses have $D=C_{\mathrm{clar}}=0$. Execution on a non-execute reference has $C_{\mathrm{call}}=0$. $\hat A_i$ is the group-normalized advantage from $R_{\mathrm{uc}}$. Thus $\alpha=4.0$, $\beta=2.5$ and $\lambda=1.0$. $W_i$ weights only the policy/GRPO loss, leaving the KL coefficient unchanged. 4. Reference-free inference. SpeechToolBench prefill computes risk from speech and supplied tools, then the policy generates a candidate. Native clarify/direct responses are retained. An execute candidate above risk 0.5 is withheld and a second pass generates clarification. Otherwise it proceeds to execution. References enter only the offline scorer after the final response is fixed.
AuthorsOpenReview ↗

Response to Reviewer dC9c (Part 3/3)

Weakness 4. The ablations do not isolate learned uncertainty from changes to action costs.

Tables 1 and 3 establish gains in accuracy, call quality and performance beyond routing alone. Section 5.4 now isolates informative risk through cost-matched controls.

Table 4. The four risk-conditioned variants share bonuses, penalties, coefficients and loss weighting. On/Off reuse a checkpoint and candidates.

Training signalRoutingAcc Norm ↑F1 ↑Tool Hall. ↓AST ↑Param ↑
Standard GRPOOff46.8263.7852.860.6430.551
Constant riskOff48.3664.5751.380.7160.585
Shuffled riskOff48.1364.2651.810.7030.579
Token entropyOff49.0264.9450.660.7420.613
Learned internal riskOff50.4266.5449.050.8120.670
Learned internal riskOn51.9166.5447.450.8030.660

Constant risk uses mean training risk, and shuffled risk permutes scores within each acoustic condition. Data, initialization, rollout budget and optimizer are matched. With routing off, learned risk exceeds constant/shuffled/entropy by 2.06/2.29/1.40 points. AST/Param use the fixed reference-execute population with non-execution scored zero, supporting better calls. Routing adds 1.49 points. F1=66.54 and Table 1's three-run mean 66.50 use the same definition.

Weakness 5. Benchmark generalization and statistical support need more detail.

Original Tables 2 and 5 (now 7) establish zero-shot transfer and three-run stability. Section 5.5 also tests policy training and the motivating diagnostic on Qwen2.5-Omni-7B.

Table 5(a). Second-backbone training. Routing is off throughout.

Training signalAcc Norm ↑AST ↑Param ↑
Standard GRPO31.820.5100.423
Constant risk33.020.5490.445
Shuffled risk32.730.5330.438
Token entropy33.470.5550.457
Learned internal risk35.130.5920.486

Table 5(b). Matched-target AUROC. Conditional failure uses correct-tool candidates. Best values are marked within each backbone.

BackboneRepresentationRouting failure AUROC ↑Conditional-call failure AUROC ↑
Qwen3-Omni-30BAudio Encoder0.9200.658
Qwen3-Omni-30BDecoder0.7060.912
Qwen2.5-Omni-7BAudio Encoder0.8740.631
Qwen2.5-Omni-7BDecoder0.6830.858

Both representations use the same held-out examples per target, with training-only probe fitting and validation selection. The 7B results replicate the diagnostic ordering and improve accuracy by 3.31 points over GRPO, supporting the design on both evaluated Qwen-Omni backbones.

Table 2. Zero-shot When2Call. Noisy averages use 5/10/15 dB. When2Call F1/Hall. use gated raw choices, and Acc uses gated length-normalized choices.

MethodNoisy AccNoisy When2Call F1Noisy Hall.Clean AccClean When2Call F1Clean Hall.
Qwen3.5-Omni-Flash70.3664.8539.5872.0066.4033.50
Qwen3.5-Omni-Plus79.5771.8330.2484.3378.7325.08
Qwen2.5-Omni-7B40.0449.9585.4444.2053.4682.11
Qwen3-Omni-30B68.7064.7342.6971.2768.2940.65
GRPO70.4465.2638.8273.4869.6236.59
UC-GRPO71.5566.1337.2376.2471.5231.71

Weights, probes and threshold transfer without When2Call tuning. UC-GRPO improves over GRPO by 1.11/2.76 accuracy points on noisy/clean audio, supporting transfer of decision preferences.

Table 17. Three matched training seeds. Mean and sample SD use unrounded accuracies.

Matched seedGRPOUC-GRPO
147.4952.42
246.2751.55
346.6951.76
Mean ± SD46.82 ± 0.6251.91 ± 0.45

All three paired gains are positive. The two-sided paired tt-test gives t(2)=50.839t(2)=50.839, p=0.0004p=0.0004, mean gain 5.09 points and 95% CI [4.66, 5.52]. This measures training-run variation on the fixed test set (Appendix E.3).

Table 19. Supplied-tool overlap with training (Appendix E.4).

SplitTool IDsSeen in TrainUnseen in Train
Validation31024862
Test30524164

Training supplies 1,420 interfaces. Identity uses function name plus canonical parameter schema, preserving validation constraints and excluding descriptions/defaults from the structural hash. All models receive identical per-request tools. Of 64 test interfaces unseen in training, 21 occur only in validation and 43 in neither split. Counts precede acoustic augmentation and describe tool exposure in a request-held-out benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.