acceptodds
Under review as a conference paper at ICLR 2027

When Should a Spoken Agent Act? A Benchmark for Risk-Aware Function Calling

Abstract

Spoken agents can send messages, control devices, and initiate payments from voice commands. Automatic speech recognition (ASR) uncertainty about a recipient or amount therefore creates an authorization problem: should the agent act, ask one question, or stop? Word error rate and function-call accuracy do not resolve that decision. We introduce Risk-Aware Spoken Function Calling (RASFC), a controlled rendered-speech benchmark that pairs structured calls with risk tiers, argument-support labels, and rule-derived next actions: execute, ask a targeted clarification, or abstain. The current benchmark interface gives its verifier the intended action's risk tier at evaluation time; this oracle context is distinct from risk inferred from speech or a predicted call. In a replayable offline diagnostic on 6,000 distinct audio/ASR observations, reference-WER plus ASR confidence misses 1,147 dialogue-required examples under a rule built from gold argument support, clarification targets, risk, and confidence, whereas a gold argument-support diagnostic misses 389. These signal comparisons measure consistency with the constructed rule. RASFC makes the information flow explicit for evaluating when uncertain speech warrants action, targeted repair, or abstention; it is a controlled benchmark interface, not an end-to-end human-speech deployment evaluation.

Primary area:
datasets and benchmarks
Keywords:
spoken function calling, automatic speech recognition uncertainty, risk-aware decision-making, action authorization, targeted clarification, abstention, rendered-speech benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_oEqQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RASFC studies whether a spoken agent should execute a func call, clarify an uncertain argument, or abstain or delay request before confirmations. The method combines structured calls, risk tiers, argument-support labels, and clarification targets with asymmetric error costs. The main replayable analysis uses 6K synthetic audio & ASR observations from 3K commands. Reference WER and ASR confidence are compared with gold-support diagnostics against a rule-based-derived dialogue target. Separate experiments study constrained callers and direct-audio baselines. The current verifier receives oracle intended-action risk, and several earlier model summaries lack replayable predictions. The paper explicitly presents its findings as controlled rule-consistency diagnostics rather than evidence of deployment safety.

Critical strengths

The execute/clarify/abstain formulation is useful. Exact-call accuracy alone misses the cost of acting on an uncertain recipient or amount. Reporting high-risk errors together with execution coverage, and treating zero executed high-risk calls as an undefined error rate, is the right evaluation discipline.

Critical weaknesses

The headline diagnostic mostly checks consistency with a rule constructed from the same information it evaluates. The dialogue target depends on gold argument support, clarification targets, confidence, and oracle risk (Appendix B). It is therefore unsurprising that gold-support features predict it better than WER buckets (Fig 2). There is no independent human validation showing that these rule-derived labels correspond to appropriate authorization or useful clarification. For a benchmark paper, this is a central validity problem.

I do not see a reproducible comparison of deployable policies at meaningful high-risk execution coverage. The verifier uses the gold intended-action risk tier; the trained waveform support-decoder code also takes annotation-derived perturbation indicators at test time, with unresolved report-to-code provenance. Every strict policy in Table 3 executes zero high-risk cases. Other nonzero-coverage results use different populations or lack the predictions and fitted policies needed for a matched replay. The released package also omits the empirical audio, weights, and per-example predictions (Sec 8).

Overall I think the method lack some innovation and insights to be accepted.

Questions for authors
  1. Can you validate a sample of execute/clarify/abstain decisions and clarification targets with independent, blinded human judgments? How often does the constructed rule disagree?
  2. Can you provide a replayable common-test comparison using inferred rather than oracle risk, without test-time perturbation annotations? Please report incorrect-call HRFE, high-risk coverage, clarification quality, and utility together.
1 comment

The headline diagnostic measures agreement with a dialogue rule built from gold clarification targets, gold argument support, confidence, and oracle risk (Appendix B). This construction makes the gold-support comparison informative about rule reconstruction, not independent validation of authorization or useful clarification. No blinded human audit or interaction outcome is reported.

The inference interface remains oracle-conditioned: the verifier receives intended-action risk, and the current support-decoder code consumes annotation-derived perturbation indicators at test time. The paper reports unresolved correspondence between that implementation and saved summaries (Sections 3 and 6.2). In Table 3, strict policies execute zero high-risk examples, so the high-risk error rate is not estimable. Separate nonzero-coverage results use different populations and do not provide a matched annotation-free operating curve.

The supplementary archive does not reproduce the empirical results because audio, weights, and per-example predictions are absent (Sections 8 and Reproducibility Statement). The current record supports a controlled benchmark diagnostic, not a validated deployed policy or positive-coverage safety result.

Reviewer_y9STOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Risk-Aware Spoken Function Calling (RASFC), a controlled rendered-speech benchmark for deciding whether a spoken agent should execute a structured function call, ask a targeted clarification, or abstain. The benchmark covers messaging, navigation, device control, and payment-style commands, pairing calls with risk tiers, argument-support labels, and rule-derived next actions. Evaluation combines call correctness, high-risk false execution, execution coverage, clarification targeting, and an asymmetric utility. The current verifier receives the intended action's gold risk tier, making the evaluation an explicitly oracle risk setting.

The principal replayable diagnostic contains 6000 distinct audio/ASR observations from 3000 source commands rendered with Apple and VITS speech. Reference WER plus ASR confidence misses 1147 rule-defined dialogue-required observations, while a diagnostic using gold argument support and clarification targets misses 389. The authors explicitly interpret these comparisons as consistency with a constructed rule. Additional experiments examine constrained and noisy-channel callers, ASR-candidate rescoring, and direct-audio approaches, with substantial limitations concerning unmatched populations, missing replay artifacts, and unresolved test-time annotation inputs.

Critical strengths

1. A consequential evaluation problem with useful outcome distinctions

The paper addresses an important limitation of evaluating spoken agents through transcription or call-prediction accuracy alone. A system's decision to execute a potentially incorrect call has consequences that differ from those of asking a focused clarification or declining to act. RASFC makes these decisions explicit and evaluates high-risk false execution alongside coverage and clarification targeting. This could give practitioners a more informative way to compare execution policies.

In particular, defining high-risk false execution over executed high-risk examples, and treating the rate as unestimable when no such example is executed, is a sound and consequential design choice (Section 3 Tables 3-4).

2. Careful disclosure of information access and evidentiary limitations

The manuscript clearly distinguishes candidate-schema risk from the oracle intended-action risk supplied during evaluation, rule consistency from independently validated authorization, and retained diagnostics from unreconstructable historical summaries. It also identifies duplicate exports and repeated candidate presentations, and avoids presenting unmatched studies as a method ranking (Table 1 Sections 6-7 and Appendix A).

These distinctions make the evidence easier to assess and help prevent misleading interpretations of low error rates or apparent sample size. I appreciate this transparency and regard it as a useful reporting example, while recognizing that the missing validation and artifacts remain substantive limitations.

Critical weaknesses

1. The principal diagnostic does not independently validate the authorization target

The principal replayable analysis establishes agreement with a rule whose defining information is supplied to the stronger diagnostic. Appendix B, lines 695-701, defines C as the existence of a gold clarification target, U as an unsupported gold argument, L as low ASR confidence, and H as high or critical intended-action risk. The dialogue-required target is

D=(C∨U∨L)∧(H∨C∨U)=C∨U∨(L∧H).D=(C\lor U\lor L)\land(H\lor C\lor U) = C\lor U\lor(L\land H).

The gold support plus target diagnostic observes the �0��0� part of this rule. Its 0.955 F1 and 389 remaining misses therefore primarily characterize recovery of the target's ingredients. The remaining misses come from the confidence and risk branch. Likewise, the source-held-out comparison between recognizer buckets and an oracle belief proxy does not resolve this dependence (Sec. 6 lines 288-310). The authors acknowledge this correctly. The substantive question is what independent evidence establishes that these targets capture appropriate authorization and useful repair.

For example, an argument lacking support in the ASR transcript need not lack support in the waveform. Conversely, matching an argument string does not by itself establish that the user has unambiguously authorized it. The current clarification metric compares against template targets, and the utility assigns a fixed reward to targeted clarification without independently establishing that an answer to the question resolves the uncertainty (Section 3, lines 157-159 and Section 7, lines 417-423). These choices can define a controlled task, but their usefulness as authorization labels still needs validation.

Risk-stratified validation is especially relevant here. Section 6 reports 364 missed high-risk dialogue required observations for WER plus confidence, compared with 389 for gold support plus target. Thus, the aggregate reduction in misses does not itself show improvement on the high-risk subset. This does not establish that the full risk aware policy is worse: the two diagnostics include different components of the constructed rule. It does reinforce the need to evaluate consequential outcomes separately from aggregate rule agreement.

To strengthen the paper, I would prioritize an independent audit of a representative, risk-stratified subset of the existing examples, with action judgments made under an explicit risk policy and without using the label generation rule as the answer key. The audit should include low-WER/high-confidence cases and disagreements between support and confidence. A controlled clarification evaluation could then test whether the selected question actually resolves the relevant argument and improves correct completion. Reporting disagreements, argument-normalization failures, and sensitivity to the confidence thresholds and utility costs would make the benchmark's intended meaning much more convincing. A controlled synthetic benchmark can be valuable; the missing element is independent support for what this particular evaluation measures.

2. The release and baseline evidence do not yet establish a reproducible empirical benchmark

According to Section 8 and the Reproducibility Statement, the supplementary archive provides a deterministic synthetic smoke test but does not reproduce the empirical summaries: the evaluated audio, trained weights, and per-example predictions are not distributed. Appendix B.1 additionally states that the earlier noisy channel result lacks its original predictions, fitted verifier, and split. This materially limits the contribution of a datasets&benchmarks submission: another researcher cannot reproduce the reported starting points or reliably compare a new policy with them from the described release.

The remaining results do not provide a sufficiently controlled replacement. Every strict operating point in Table 3 executes zero high-risk examples, so those points do not characterize selective execution at positive high-risk coverage. The paper correctly reports the corresponding error rates as unestimable. There are other results with nonzero high-risk coverage, including 39/155 and 111/465 in Section 6.2, but they use different evaluated populations. In addition, the current supervised support-decoder implementation consumes annotation-derived perturbation indicators at test time, and the saved summaries have not been matched to that implementation (Table 1 Section 6.2, lines 393-406 and Table 13). This unresolved correspondence prevents determining which inputs produced those results or isolating the contribution of waveform evidence.

The oracle intended-action risk tier is a clearly disclosed controlled condition and can be retained as such. The immediate concern is the absence of a complete comparison with verified inputs, common candidates and splits, and interpretable execution outcomes even within that condition. Similarly, the acknowledged reuse of some waveforms in multiple candidate modes requires source-group accounting. Those rows should not be treated as independent recordings.

To improve the paper, I recommend consolidating the empirical section around one versioned cohort and a compact matched comparison: a transcript/confidence baseline, an argument support or ASR candidate method, and a waveform method if its results are retained. Use identical source disjoint partitions, legal candidate inventories, risk information, and threshold selection rules. Release the evaluated observations or a complete regeneration route, labels, source and split identifiers, fitted-model or configuration bindings, and per example outputs supporting the main table. Report actual incorrect-call high risk error against execution coverage, clarification outcomes, utility, and uncertainty computed at the source-command level. This would provide substantially stronger evidence than additional unmatched model summaries. A candidate-derived risk condition would further clarify practical applicability, while the oracle condition remains clearly labeled.

Questions for authors

The following questions concern evidence that could change my assessment. Broader evaluation on natural human speech would be a useful extension. The immediate requests focus on the validity and reproducibility of the submitted benchmark.

  • Label construction and independent validation. How exactly are gold argument support and clarification recoverability determined, including aliases, number normalization, and correct but nonliteral transcripts? What evidence independent of the constructed dialogue rule supports the execute/clarify/abstain labels? Please identify any existing adjudication or outcome-based validation and its protocol.

  • Availability of the central benchmark. Can the authors provide a complete replayable package for one principal cohort, including audio or regeneration assets, labels, candidate inventories, source-disjoint splits, and the configurations and predictions needed to recover a main comparison? Which reported results can currently be regenerated, given the limitations stated in Section 8 and App. B.1?

  • Inputs behind the support-decoder results. Which exact code, model checkpoint, and test-time inputs produced the reported nonzero high-risk coverage? Were perturbation annotations supplied in those runs? A matched comparison excluding those annotations would materially clarify whether the results measure recoverable waveform evidence.

  • Evidence at nonzero high-risk coverage. On a shared held-out cohort and information interface, what tradeoff does the strongest replayable policy achieve between execution coverage and actual incorrect call high-risk error? Can the authors provide a matched operating point or curve, with source group uncertainty, that distinguishes this outcome from agreement with the dialogue rule or cross renderer disagreement?

3 comments

The label audit and benchmark replay are not independent in the submitted version. Appendix B defines the dialogue-required target from gold clarification targets, unsupported gold arguments, low ASR confidence, and oracle risk; the paper reports no separate adjudication or outcome-based validation of these labels. It reports synthetic rendered speech rather than human recordings (Abstract, p. 1; Section 8, p. 9).

The released archive supplies a deterministic synthetic smoke test, but the empirical audio, trained weights, and per-example predictions needed to reproduce the summaries are not distributed (Sections 8 and Reproducibility Statement, pp. 9–10). The submission does not identify a complete replayable principal cohort.

The paper discloses that the current support-decoder implementation uses annotation-derived perturbation indicators at test time, while saved summaries have not been matched to that implementation (Sections 3 and 6.2). Strict policies execute zero high-risk examples, so HRFE is not estimable; other nonzero-coverage summaries use different populations and unresolved bindings (Tables 3–4, pp. 6–8). No matched annotation-free comparison with inferred risk and source-group uncertainty is reported.

Reviewer_y9STOpenReview ↗

Official comment to authors

Thank you for your response. Unfortunately, my concerns have not been resolved and I will keep my original score.

Thank you for the follow-up. We added a source-grouped held-out sensitivity to test whether the support-head result depends on annotation-derived inputs. In the stored eSpeak yes/no split, 1,958/3,000 upstream commands had realizations across train, development, and test. We froze the source groups before refitting matched heads (2,100/450/450 groups and 6,300/1,350/1,350 examples); the test set contains 525 annotated high-risk examples.

At the development-selected cap targeting zero HRFE, the full-feature head covered 162/525 high-risk cases (30.9%, source-group bootstrap 95% CI [24.3%, 37.1%]) with 5 errors (HRFE 0.95%, CI [0.18%, 2.12%]). Removing annotated-risk, candidate score/confidence, dataset/renderer/mode, perturbation, and candidate-decision inputs reduced coverage to 115/525 (21.9%, CI [16.3%, 27.6%]) with 4 errors (HRFE 0.76%, CI [0.18%, 1.59%]). Paired two-action utility (ablated minus full) was -0.130 (CI [-0.169, -0.087]); the paired HRFE difference was -0.19% (CI [-1.25%, 0.75%]), which spans zero. The ablation retains the schema-derived risk tier (99.85% agreement with the annotated tier). This gives a source-separated, nonzero-coverage sensitivity result while leaving the paired high-risk error difference unresolved.

The revision does not establish independent actionability: it includes no blinded human or outcome audit, risk remains schema-supplied, candidate generation is transcript-conditioned and risk-prompted, and source-group predictions were unavailable for the headline 0.437 utility estimate. We added the sensitivity and these limits in Discussion and Limitations (p. 8) and Supplement, Section “Source-Grouped Waveform-Support Sensitivity.” We respectfully submit that this narrows the decoder-specific concern, while independent label validity remains open.

Reviewer_gZ53OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RASFC, a benchmark for deciding whether a spoken agent should execute, clarify one argument, or abstain under ASR uncertainty. Each example pairs a TTS-rendered command with a gold function call, a risk tier, argument-support labels, and a clarification target. The main diagnostic uses 3,000 commands rendered with two TTS systems (6,000 observations) and argues that reference WER plus ASR confidence fails to identify dialogue-required turns, whereas gold argument support does. The paper also reports a noisy-channel constrained caller with a risk verifier and several direct-audio comparators.

Critical strengths
  • The problem (execution authorization under ASR uncertainty, rather than transcript accuracy) is relevant and well motivated.
  • Separating call correctness, risk tier, and clarification target is a sensible label design, and reporting the high-risk execution denominator - alongside HRFE is good practice.
Critical weaknesses
  • No independent validation of the actionability target. The dialogue-required target is generated from gold clarification targets, gold argument support, confidence, and risk. Figure 2 and Table 2 then evaluate diagnostics that include these same gold signals. Their strong agreement therefore partly reflects the target construction, rather than independently establishing when clarification is necessary. The paper acknowledges this dependence, but provides neither human validation of the risk and clarification labels nor an evaluation showing that clarification improves actual interaction outcomes. All speech is synthetic, further limiting evidence beyond the controlled setting.

  • Evaluation depends on oracle information. The verifier and tier-specific threshold policy receive the gold risk tier of the intended action at test time, rather than a risk estimate derived from the observed speech or predicted call. Candidate-derived risk policies are not evaluated. In addition, the current support-decoder implementation consumes annotation-derived perturbation indicators at test time, while the correspondence between that implementation and the reported decoder results remains unverified. These results therefore do not establish performance under an inference setting without benchmark annotations.

  • The strict-policy results do not demonstrate high-risk selective execution. Every strict policy in Table 3 executes zero high-risk examples, making HRFE unestimable at those operating points. Although the paper correctly reports this limitation, these results do not show that the policies can execute high-risk commands while controlling errors. Separate support-head results have nonzero high-risk coverage, but the paper states that unresolved annotation-input bindings prevent those summaries from isolating waveform-based support.

Questions for authors

see weakness

1 comment

RASFC is a controlled rendered-speech benchmark. Its dialogue-required target is constructed from gold clarification targets, gold argument support, ASR confidence, and oracle intended-action risk; the gold-support diagnostic uses information that overlaps this target (Sections 2–3 and Appendix B). The reported 0.955 F1 and 389 remaining misses therefore measure agreement with the constructed rule. They do not independently validate when clarification is appropriate or whether it improves an interaction. The paper reports no independent label audit or user-outcome study.

The verifier and tier-specific policy receive the gold intended-action risk tier. The support-decoder implementation also consumes annotation-derived perturbation indicators at test time, and the paper says its saved summaries are not yet bound to that implementation (Sections 3 and 6.2). The strict Table 3 policies execute zero high-risk examples, making HRFE unestimable at those points (p. 6). These results do not establish high-risk selective execution or an annotation-free inference result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.