- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes UARI for training tool-calling LLMs. It generates actions from interrupted reasoning prefixes, scores their action-token likelihoods, selects a reward-diverse group within a median confidence window, and combines correctness advantages with bounded relative-confidence shaping. Diagnostics motivate using interrupted action confidence rather than full-trace likelihood. On Qwen3-4B, mean BFCL accuracy rises from 68.85% with Plain GRPO to 70.05%, and single-turn Brier score decreases from 0.1514 to 0.1494. The paper also evaluates Llama and Granite backbones and Nestful, and studies the contributions of interruption, selection, and shaping.
Critical strengths
-
The paper asks a relevant question about whether reasoning-token likelihood is an appropriate confidence signal for structured actions. The step-swap and held-out residual diagnostics (Appendix B) provide useful evidence that narrative confidence can vary without comparable changes in action correctness. The manuscript appropriately acknowledges that residualization is a limited linear diagnostic, not causal identification. Using regenerated actions to probe intermediate reasoning states is an interesting way to obtain candidate-level training signals.
-
The component comparisons are informative. Table 3 holds the interrupted candidate pool and update budget fixed for random, trace-guided, and action-guided selection. Action-guided selection reaches 36.25% multi-turn accuracy versus 32.25% for trace guidance, and shaping adds a further 0.50 points. Random selection and shaping alone do not improve mean accuracy over Plain GRPO. These negative controls make the observed benefit of the combined procedure more credible than an isolated benchmark gain. Evaluation uses standard inference rather than requiring interrupted sampling at deployment.
Critical weaknesses
-
The evidence supports useful candidate ranking more clearly than the central calibration claim. Equation 6 rewards group-relative action likelihood; it does not directly compare a predicted success probability with correctness. Clipping bounds that contribution but does not itself establish calibration. Appendix D uses exp(mean token log-probability) as the success-probability estimate; this is a geometric mean token likelihood and needs empirical validation as a correctness probability. Brier is a valid proper score, but its improvement can reflect changes in accuracy/discrimination as well as calibration. Against Plain GRPO, the single-turn Brier improvement is only 0.0020, and Non-Live AUROC decreases from 66.81% to 63.71% (Table 2). No uncertainty estimates are reported, and the extensive component ablations report accuracy rather than calibration. Thus the manuscript does not yet establish that the proposed shaping improves calibration, rather than primarily improving action selection and accuracy.
-
The practical benefit over standard RL is not evaluated at a matched rollout budget. Appendix F.2 matches data, update group size, and training schedule, which is useful, but UARI generates four full source traces plus regenerated actions at multiple prefixes, whereas Plain GRPO directly generates six traces. The shared-pool ablations isolate the confidence signal within UARI, but do not resolve whether its relatively modest gains over GRPO persist under equal total sampling compute. The candidate-count distribution, total generated-token/compute costs, and an equally resourced GRPO or reward-diversity-selection comparison are missing. Together with the calibration evidence gap, this leaves the overall benefit insufficiently established for acceptance in its present form.
Questions for authors
-
For action-guided selection without shaping and for full UARI, can you report Brier/AUROC with uncertainty estimates and a reliability analysis separating calibration from discrimination? How many training seeds and evaluation samples underlie Table 2? This would directly test whether Equation 6 contributes the calibration benefit claimed.
-
What are the observed interrupted-pool sizes and total rollout tokens or wall-clock costs relative to Plain GRPO? Does UARI retain its benefit when the comparison matches sampling compute rather than only the six selected update candidates?
-
What validation split and criterion select the confidence weight and training checkpoint? Clarifying whether these choices were fixed independently of the reported BFCL test results would help interpret the small differences and the lambda sensitivity in Table 5.