1. The principal diagnostic does not independently validate the authorization target
The principal replayable analysis establishes agreement with a rule whose defining information is supplied to the stronger diagnostic. Appendix B, lines 695-701, defines C as the existence of a gold clarification target, U as an unsupported gold argument, L as low ASR confidence, and H as high or critical intended-action risk. The dialogue-required target is
D=(C∨U∨L)∧(H∨C∨U)=C∨U∨(L∧H).
The gold support plus target diagnostic observes the �0� part of this rule. Its 0.955 F1 and 389 remaining misses therefore primarily characterize recovery of the target's ingredients. The remaining misses come from the confidence and risk branch. Likewise, the source-held-out comparison between recognizer buckets and an oracle belief proxy does not resolve this dependence (Sec. 6 lines 288-310). The authors acknowledge this correctly. The substantive question is what independent evidence establishes that these targets capture appropriate authorization and useful repair.
For example, an argument lacking support in the ASR transcript need not lack support in the waveform. Conversely, matching an argument string does not by itself establish that the user has unambiguously authorized it. The current clarification metric compares against template targets, and the utility assigns a fixed reward to targeted clarification without independently establishing that an answer to the question resolves the uncertainty (Section 3, lines 157-159 and Section 7, lines 417-423). These choices can define a controlled task, but their usefulness as authorization labels still needs validation.
Risk-stratified validation is especially relevant here. Section 6 reports 364 missed high-risk dialogue required observations for WER plus confidence, compared with 389 for gold support plus target. Thus, the aggregate reduction in misses does not itself show improvement on the high-risk subset. This does not establish that the full risk aware policy is worse: the two diagnostics include different components of the constructed rule. It does reinforce the need to evaluate consequential outcomes separately from aggregate rule agreement.
To strengthen the paper, I would prioritize an independent audit of a representative, risk-stratified subset of the existing examples, with action judgments made under an explicit risk policy and without using the label generation rule as the answer key. The audit should include low-WER/high-confidence cases and disagreements between support and confidence. A controlled clarification evaluation could then test whether the selected question actually resolves the relevant argument and improves correct completion. Reporting disagreements, argument-normalization failures, and sensitivity to the confidence thresholds and utility costs would make the benchmark's intended meaning much more convincing. A controlled synthetic benchmark can be valuable; the missing element is independent support for what this particular evaluation measures.
2. The release and baseline evidence do not yet establish a reproducible empirical benchmark
According to Section 8 and the Reproducibility Statement, the supplementary archive provides a deterministic synthetic smoke test but does not reproduce the empirical summaries: the evaluated audio, trained weights, and per-example predictions are not distributed. Appendix B.1 additionally states that the earlier noisy channel result lacks its original predictions, fitted verifier, and split. This materially limits the contribution of a datasets&benchmarks submission: another researcher cannot reproduce the reported starting points or reliably compare a new policy with them from the described release.
The remaining results do not provide a sufficiently controlled replacement. Every strict operating point in Table 3 executes zero high-risk examples, so those points do not characterize selective execution at positive high-risk coverage. The paper correctly reports the corresponding error rates as unestimable. There are other results with nonzero high-risk coverage, including 39/155 and 111/465 in Section 6.2, but they use different evaluated populations. In addition, the current supervised support-decoder implementation consumes annotation-derived perturbation indicators at test time, and the saved summaries have not been matched to that implementation (Table 1 Section 6.2, lines 393-406 and Table 13). This unresolved correspondence prevents determining which inputs produced those results or isolating the contribution of waveform evidence.
The oracle intended-action risk tier is a clearly disclosed controlled condition and can be retained as such. The immediate concern is the absence of a complete comparison with verified inputs, common candidates and splits, and interpretable execution outcomes even within that condition. Similarly, the acknowledged reuse of some waveforms in multiple candidate modes requires source-group accounting. Those rows should not be treated as independent recordings.
To improve the paper, I recommend consolidating the empirical section around one versioned cohort and a compact matched comparison: a transcript/confidence baseline, an argument support or ASR candidate method, and a waveform method if its results are retained. Use identical source disjoint partitions, legal candidate inventories, risk information, and threshold selection rules. Release the evaluated observations or a complete regeneration route, labels, source and split identifiers, fitted-model or configuration bindings, and per example outputs supporting the main table. Report actual incorrect-call high risk error against execution coverage, clarification outcomes, utility, and uncertainty computed at the source-command level. This would provide substantially stronger evidence than additional unmatched model summaries. A candidate-derived risk condition would further clarify practical applicability, while the oracle condition remains clearly labeled.
The headline diagnostic measures agreement with a dialogue rule built from gold clarification targets, gold argument support, confidence, and oracle risk (Appendix B). This construction makes the gold-support comparison informative about rule reconstruction, not independent validation of authorization or useful clarification. No blinded human audit or interaction outcome is reported.
The inference interface remains oracle-conditioned: the verifier receives intended-action risk, and the current support-decoder code consumes annotation-derived perturbation indicators at test time. The paper reports unresolved correspondence between that implementation and saved summaries (Sections 3 and 6.2). In Table 3, strict policies execute zero high-risk examples, so the high-risk error rate is not estimable. Separate nonzero-coverage results use different populations and do not provide a matched annotation-free operating curve.
The supplementary archive does not reproduce the empirical results because audio, weights, and per-example predictions are absent (Sections 8 and Reproducibility Statement). The current record supports a controlled benchmark diagnostic, not a validated deployed policy or positive-coverage safety result.