acceptodds
Under review as a conference paper at ICLR 2027

Eliciting Action Uncertainty via Reasoning Interruption for Training Calibrated Tool-Using LLMs

Abstract

Reinforcement learning improves the execution accuracy of tool-using LLMs but often leaves them poorly calibrated, causing the models to remain overconfident in incorrect actions and thereby undermining their reliability and trustworthiness. Existing approaches inject a calibration term into the optimization objective using uncertainty estimates aggregated over the entire reasoning-to-action sequence. Such trace-level estimates conflate linguistic fluency with action correctness: because action-relevant information is sparse, the resulting uncertainty signal is diluted by generic reasoning tokens. To recover an action-faithful uncertainty signal, we interrupt the reasoning trace at step boundaries and regenerate tool calls from each prefix. Our analysis shows that the action's log-likelihood, evaluated under such interrupted prefixes, ranks correct tool calls above failures more reliably than its full-trace counterpart. Based on this observation, we propose UARI (Uncertainty-aware Reasoning Interruption), an RL framework that turns this interrupted-action signal into an uncertainty-aware training objective: candidates are grouped by uncertainty-induced difficulty and the same signal supplies a bounded reward-shaping term, jointly optimizing execution correctness and calibration. By targeting an uncertainty signal decoupled from generic reasoning tokens, UARI addresses miscalibration at the specific juncture of action commitment. Empirical evaluations on the BFCL-v4 and Nestful benchmarks demonstrate that UARI consistently outperforms the standard GRPO baseline in both execution accuracy and the calibration peromance.

Primary area:
reinforcement learning
Keywords:
tool use, function calling, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9oy7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UARI for training tool-calling LLMs. It generates actions from interrupted reasoning prefixes, scores their action-token likelihoods, selects a reward-diverse group within a median confidence window, and combines correctness advantages with bounded relative-confidence shaping. Diagnostics motivate using interrupted action confidence rather than full-trace likelihood. On Qwen3-4B, mean BFCL accuracy rises from 68.85% with Plain GRPO to 70.05%, and single-turn Brier score decreases from 0.1514 to 0.1494. The paper also evaluates Llama and Granite backbones and Nestful, and studies the contributions of interruption, selection, and shaping.

Critical strengths
  1. The paper asks a relevant question about whether reasoning-token likelihood is an appropriate confidence signal for structured actions. The step-swap and held-out residual diagnostics (Appendix B) provide useful evidence that narrative confidence can vary without comparable changes in action correctness. The manuscript appropriately acknowledges that residualization is a limited linear diagnostic, not causal identification. Using regenerated actions to probe intermediate reasoning states is an interesting way to obtain candidate-level training signals.

  2. The component comparisons are informative. Table 3 holds the interrupted candidate pool and update budget fixed for random, trace-guided, and action-guided selection. Action-guided selection reaches 36.25% multi-turn accuracy versus 32.25% for trace guidance, and shaping adds a further 0.50 points. Random selection and shaping alone do not improve mean accuracy over Plain GRPO. These negative controls make the observed benefit of the combined procedure more credible than an isolated benchmark gain. Evaluation uses standard inference rather than requiring interrupted sampling at deployment.

Critical weaknesses
  1. The evidence supports useful candidate ranking more clearly than the central calibration claim. Equation 6 rewards group-relative action likelihood; it does not directly compare a predicted success probability with correctness. Clipping bounds that contribution but does not itself establish calibration. Appendix D uses exp(mean token log-probability) as the success-probability estimate; this is a geometric mean token likelihood and needs empirical validation as a correctness probability. Brier is a valid proper score, but its improvement can reflect changes in accuracy/discrimination as well as calibration. Against Plain GRPO, the single-turn Brier improvement is only 0.0020, and Non-Live AUROC decreases from 66.81% to 63.71% (Table 2). No uncertainty estimates are reported, and the extensive component ablations report accuracy rather than calibration. Thus the manuscript does not yet establish that the proposed shaping improves calibration, rather than primarily improving action selection and accuracy.

  2. The practical benefit over standard RL is not evaluated at a matched rollout budget. Appendix F.2 matches data, update group size, and training schedule, which is useful, but UARI generates four full source traces plus regenerated actions at multiple prefixes, whereas Plain GRPO directly generates six traces. The shared-pool ablations isolate the confidence signal within UARI, but do not resolve whether its relatively modest gains over GRPO persist under equal total sampling compute. The candidate-count distribution, total generated-token/compute costs, and an equally resourced GRPO or reward-diversity-selection comparison are missing. Together with the calibration evidence gap, this leaves the overall benefit insufficiently established for acceptance in its present form.

Questions for authors
  1. For action-guided selection without shaping and for full UARI, can you report Brier/AUROC with uncertainty estimates and a reliability analysis separating calibration from discrimination? How many training seeds and evaluation samples underlie Table 2? This would directly test whether Equation 6 contributes the calibration benefit claimed.

  2. What are the observed interrupted-pool sizes and total rollout tokens or wall-clock costs relative to Plain GRPO? Does UARI retain its benefit when the comparison matches sampling compute rather than only the six selected update candidates?

  3. What validation split and criterion select the confidence weight and training checkpoint? Clarifying whether these choices were fixed independently of the reported BFCL test results would help interpret the small differences and the lambda sensitivity in Table 5.

Reviewer_NkdYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that full-trace confidence is a poor signal for tool-call correctness, because most reasoning tokens are near-certain filler. It truncates reasoning at entropy-selected sentence boundaries, regenerates the tool call from each prefix, and uses the mean log-likelihood of the regenerated action as confidence. UARI builds a candidate pool from these regenerations, keeps a median-centered window ranked by this confidence, picks the 6 candidates with the highest reward variance as the GRPO group, and adds a small confidence term to the advantage (lambda=0.05). On Qwen3-4B trained on ToolRL data, it reports gains over plain GRPO on BFCL-v4 (70.05 vs 68.85 mean) and Nestful (11.9 vs 10.4), plus better AUROC/Brier than other calibration-oriented RL methods.

Critical strengths

The observation that tool-call uncertainty sits in a few action tokens, and that averaging over the whole trace dilutes it, is reasonable and probably useful. Using truncated-prefix regeneration to build GRPO candidates is a new idea.

Critical weaknesses
  1. The shaping term does not target calibration. In Eq. 6 the confidence advantage is the group-normalized confidence itself, with no dependence on correctness, so a confident wrong action within a mixed group still gets a bonus. That is closer to entropy minimization or a self-certainty reward than to a calibration objective, and in principle it pushes toward the overconfidence the paper sets out to fix. The authors effectively acknowledge this by turning the term off for all-zero groups. A calibration objective would reward confidence only when the action is correct (e.g. a Brier- or log-loss-style term). No calibration numbers are reported for lambda=0, so we can't tell whether the term does anything for calibration.

  2. The calibration and accuracy gains over plain GRPO are small and not clearly attributable. In Table 2, UARI's Non-Live AUROC is lower than GRPO's (63.71 vs 66.81), and Brier differences are 0.1-0.3, which accuracy changes alone could produce. The large margins over CAR and DCPO come from those methods sitting near AUROC 50, likely because they train verbalized confidence but are evaluated with log-probability confidence. Brier uses exp(mean token log-prob) as if it were P(correct), and there is no ECE, no reliability plot, and no selective-prediction experiment even though deferral is the stated motivation. On accuracy, the gain is about 1 point, from a single seed, and the comparison is not compute-matched (4 full rollouts plus many regenerations vs 6 rollouts for GRPO). Picking the max-reward-variance subset is a form of dynamic sampling that plain GRPO doesn't get. The training data is single-step, yet most of the gain is on multi-turn (about 8 tasks out of 200). Reasoning also gets shorter, which could explain part of the effect.

Questions for authors
  1. Under a matched generation budget, or with max-variance group selection given to GRPO as well, does the accuracy gain remain?
  2. Results over multiple seeds?
Reviewer_GhpjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies confidence estimation and calibration in tool-using LLMs. The proposed UARI framework interrupts reasoning at selected sentence boundaries and regenerates tool calls from the retained prefixes. It uses action confidence to filter candidates into a median-centered window, selects a group maximizing correctness-reward variance, and combines correctness advantages with bounded confidence shaping for GRPO updates. Experiments on BFCL-v4 and Nestful evaluate execution accuracy, confidence ranking, and Brier score, alongside component ablations.

Critical strengths
  1. The paper addresses a practically relevant problem, since improved tool-call accuracy does not necessarily imply reliable confidence estimates. The prefix and perturbation diagnostics motivate studying action-level uncertainty.
  2. UARI improves mean BFCL-v4 accuracy over Plain GRPO on both Qwen3-4B and Llama-3.2-3B, and improves Qwen3-4B Nestful accuracy without additional inference calls (Table 1).
  3. The ablations compare trace-guided and action-guided selection using the same interrupted candidate pool, providing evidence that the confidence signal matters beyond simply enlarging the pool (Table 3).
Critical weaknesses
  1. The novelty relative to closely related training methods is not adequately established. S-GRPO (Dai et al., NeurIPS 2025) already constructs RL groups by generating answers from multiple truncated reasoning prefixes. Step-GRPO (Chen et al., 2026) uses induced-answer confidence to control truncation during GRPO rollouts. Both target reasoning efficiency, whereas UARI targets tool-call calibration through confidence-guided grouping and shaping. However, neither work is discussed or compared, leaving the additional contribution beyond existing truncation-based RL insufficiently established.
  2. The independent benefit of interruption remains unclear. Figure 1 includes a completed-prefix diagnostic, but Table 3 lacks an end-to-end comparison using action-only confidence after full reasoning with the same selection and shaping rules. A reward-variance-only selection baseline is also missing. The existing ablations support the combined method, but do not fully distinguish the value of interruption from action-only scoring and reward-based group construction.
  3. The connection between confidence shaping and calibration is insufficiently demonstrated. Correctness remains the dominant signal in Equation (6), but the confidence component can reduce the penalty on a high-confidence incorrect call; it does not directly reward agreement between confidence and correctness. In Table 2, pooled Brier improves only from 15.14 to 14.94 versus Plain GRPO on the reported scale, while Non-Live AUROC falls from 66.81 to 63.71. AUROC measures ranking, and Brier reflects both calibration and discrimination. Reliability diagrams and calibration-focused component ablations are needed to substantiate the central claim.
Questions for authors
  1. How does UARI differ from S-GRPO and Step-GRPO, and how does it compare with their adaptations to tool calling under matched generation budgets?
  2. Can the authors test full-reasoning action-only confidence with the same selection and shaping rules, and reward-variance-only selection from the interrupted candidate pool?
  3. Can reliability diagrams and calibration ablations for selection-only versus full UARI clarify the confidence component's effect, particularly on high-confidence incorrect calls?
  4. Why are all Llama-3.2-3B Nestful results omitted from Table 1? No specific reason is provided in the text or appendix. Can at least Plain GRPO and UARI be evaluated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.