acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Verbalized Confidence at Any Output Token Position

Abstract

Large language models often report confidence that does not match their accuracy, and existing methods typically calibrate it at the before-answer (BA) or after-answer (AA) position. This leaves open what confidence denotes at an arbitrary output position and how it should be calibrated. We define the target at token as the conditional probability of eventual answer correctness given the current prefix; these values form a Doob martingale whose endpoints are BA and AA confidence. We then show empirically that coupling a BA report with answer generation can alter subsequent accuracy, so such reports need not estimate the solver's success probability under a plain reasoning prompt. Position-Agnostic Confidence Elicitation (PACE) therefore decouples the two: it samples and grades solutions once, then trains a confidence readout at randomly selected prefixes with a Brier reward, which we prove is uniquely maximized by the conditional correctness probability for a fixed solver and position-selection rule. We further analyze how group normalization in group relative policy optimization (GRPO) alters this objective. Mean subtraction preserves the expected reward-gradient direction, whereas standardization yields a comparison semi-gradient that, with two samples, induces threshold-type best responses; we characterize larger groups through structural identities and ablations. Across five backbones and seven test sets, PACE matches or outperforms strong baselines designed for one or two positions. Finally, confidence-based routing demonstrates the value of position flexibility: deferring at BA saves more cost than routing on AA confidence.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Confidence Calibration, Reinforcement Learning, Verbalized Confidence, Position-Agnostic, Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_P3pmOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to calibrate an LLM's verbalized confidence at any point during generation, not just before or after the answer. The target at each position is the probability that the final answer will be correct, given the partial solution so far. The authors first show that asking for confidence before answering can change the answer itself. Motivated by this, they propose PACE. PACE fixes a set of graded solutions and trains the model to report its confidence at random cut points using a Brier reward. They also show that GRPO's standard-deviation normalization hurts calibration. Across five models and seven benchmarks, PACE is comparable to or better than existing methods at both ends. A routing experiment shows that deciding before the answer saves more cost.

Critical strengths
  1. A simple, well-motivated design. The paper shows that asking for confidence before answering can change the answer itself, so a confidence trained this way may not reflect what the model would actually do. Training confidence on fixed, already-graded solutions avoids this problem cleanly, and one trained model can then report confidence at any position.

  2. Careful and honest evaluation. The method is tested across several models and benchmarks, and the ablations line up with the main design choices, especially the effect of GRPO's normalization. The authors are also clear about what their theory does and does not cover.

Critical weaknesses

None

Questions for authors

How does PACE perform at fixed token positions? Intermediate positions are chosen as a fraction of the final solution length, but during real generation that length is unknown. How will the author solve this problem?

3 comments

Thank you for the positive assessment and for a question that matters for deployment.

Q: The final length is unknown during generation. How does PACE perform at fixed token positions? PACE never needs the final length. The probe state st=(x,R,y≤t,P)s_t=(x,R,y_{\le t},P) contains only the prefix; the model never sees uu or TT. The relative position uu is only a sampling rule for choosing training prefixes, and at inference one probes at any absolute index: every kk tokens, at a step boundary, or on demand. The revision states this in a Deployment paragraph (Section 5).

To test it directly, we probed at fixed token indices, each model on its own DeepMath-500 solutions. At index tt only solutions longer than tt remain, and their accuracy falls with tt because long solutions are more often wrong, so a correct readout must track a moving base rate. As in Table 45, surv./acc. is the fraction of solutions longer than tt and their accuracy:

tt (tokens)Base-Verb: surv. / acc.ECE ↓AUROC ↑Brier ↓Base-PACE-uu: surv. / acc.ECE ↓AUROC ↑Brier ↓
0100% / 0.630.3510.5580.355100% / 0.640.0740.7450.194
25699% / 0.630.3500.6050.35299% / 0.640.0660.7490.194
51292% / 0.610.3720.5980.37292% / 0.620.0690.7390.201
102455% / 0.520.4590.5960.45657% / 0.520.0740.6950.224
153631% / 0.440.5320.5980.52632% / 0.420.0850.6630.229
204818% / 0.370.6030.6300.59219% / 0.360.1080.6680.221
30727% / 0.170.8000.5920.7817% / 0.190.2320.6190.204
tt (tokens)RLVR-Verb: surv. / acc.ECE ↓AUROC ↑Brier ↓RLVR-PACE-uu: surv. / acc.ECE ↓AUROC ↑Brier ↓
0100% / 0.800.1970.5990.199100% / 0.810.0350.7460.133
256100% / 0.800.1930.6320.19599% / 0.810.0310.7750.128
51294% / 0.790.2060.6190.20793% / 0.800.0330.7840.132
102462% / 0.720.2680.6230.26862% / 0.730.0470.7650.164
153637% / 0.630.3660.6040.36438% / 0.640.0560.7430.194
204822% / 0.500.4830.6080.47924% / 0.540.0810.7120.222
30729% / 0.260.7170.5710.70710% / 0.350.1820.6370.249
  • Through 1024 tokens, PACE is as well calibrated at fixed indices as at relative positions (ECE at most 0.074 and 0.047), and its confidence follows the falling accuracy of the remaining solutions. The untrained models report about 0.97–0.98 at every index, so their ECE grows as accuracy falls.
  • Beyond 2048 tokens PACE becomes over-confident (ECE 0.23 and 0.18 at 3072 tokens, where 7–10% of solutions remain). We attribute this to the training data: under uniform uu a solution of length TT is cut beyond token KK with probability max⁡(0,1−K/T)\max(0,1-K/T), so only about 5% of training probe states lie beyond 2048 tokens and about 1% beyond 3072, and those long prefixes are mostly incorrect.

This table is Table 45 (Appendix J.3) in the revision, and the Limitations section states the degradation.

Thank you for the detail rebuttal. I would keep my positive score.

Thank you for taking the time to read our rebuttal and for confirming your positive assessment. We sincerely appreciate your thoughtful review and consideration.

Reviewer_b85vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates probabilistic calibration not just after answer generation or before, but at arbitrary positions along the reasoning trace.

Parts of the solution to achieve this is fairly straightforward: Sample solutions, introduce random cut-off points, elicit verbalized confidence.

However, the GPRO training with custom Brier reward seems to have nice properties.

Critical strengths

Problem setting potentially novel: Although calibration in reasoning settings has been studied a lot, the framing here might be novel.

The Brier reward appears to have some nice theoretical properties.

Critical weaknesses

I wonder what the purpose of Section 4 is. See question below.

The before-answer approach was introduced by Chen et al. "Query-level uncertainty in large language models" (arxiv 2025 / ICLR 2026), not by the later CoCA paper (which cites Chen et al.).

Questions for authors

I wonder what the purpose of Section 4 is: Inserting a confidence into the prompt is obviously going to distort the subsequent reasoning substantially. In practice, why would we leave this confidence as part of the prefix when decoding? Why not backtrack and decode without it?

1 comment

Thank you for the review and the two concrete points.

Q: Why not backtrack and decode without the confidence? What is Section 4 for? We agree that backtracking is the right design, and it is exactly what PACE does: confidence is elicited in a separate probe state st=(x,R,y≤t,P)s_t=(x,R,y_{\le t},P) whose output never enters the answer (Section 3, Figure 2). Section 4 explains why this must hold during training, which is where prior BA methods differ:

  • The BA baselines do not backtrack. CoCA, CALIBER's pre-confidence and coupled BA prompts emit the confidence and then the answer in one rollout, and reward the confidence against the correctness of that answer.
  • Section 4 shows that this target is contaminated. Forcing the announced BA value from 0 to 1 raises pass@1 by 0.21 (Qwen3-8B), 0.06 (Llama-3.1-8B) and 0.19 (GLM-4-9B), with a monotone dose response. A coupled BA confidence is therefore calibrated to an accuracy that its own announcement shaped, not to the model's success probability p(x)p(x) under a plain prompt. Jang et al. (ICML 2026) observe the same effect after SFT.
  • Backtracking at inference does not repair such a model. Its confidence was trained to predict accuracy given its own announcement; once the answer is decoded without the announcement, the confidence targets a different quantity from the one it learned. PACE avoids the mismatch in training as well, because its reward never depends on an answer generated after the report.
  • Decoupling is cheap. The probe is appended to the already-cached prefix, so a confidence costs a short prompt and about six decoded tokens (the numeral and its closing brace); the reasoning is never re-decoded.

Section 4 now opens with the following paragraph to explain its role:

Coupled methods emit the confidence and the answer in one rollout and reward the confidence against the answer that follows it. We find that this target is contaminated: the announced confidence can change answer correctness, especially for BA reports. This motivates eliciting confidence in a separate probe whose output is never fed back into the answer, the design we adopt in Section 5.

W2: Attribution of BA confidence. You are right. Query-level (BA) uncertainty was introduced by Chen et al., "Query-Level Uncertainty in Large Language Models" (ICLR 2026), with a training-free internal-state estimator; CoCA later trains a verbalized BA confidence. We corrected the attribution in the Introduction and Section 2. Query-level uncertainty estimates V0=p(x)V_0=p(x), the BA endpoint of our target.

Reviewer_ic7zOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that prior research on language model confidence calibration has primarily considered confidence elicitation at either the before-answer (BA) or after-answer (AA) position, overlooking intermediate positions during generation. To address this limitation, the authors aim to make a calibration method that estimates the conditional probability of eventual answer correctness given the current prompt at an arbitrary position tt.

The paper also show that eliciting confidence at the BA position can influence subsequent answer accuracy. Based on these observations, they propose position-agnostic confidence elicitation (PACE). Specifically, their approach (1) generates multiple reasoning trajectories for each question and obtains verified correctness labels, and (2) truncates each trajectory at a randomly selected position, prompts the LLM to elicit its confidence, and computes a Brier-score-based reward using the final correctness label.

Furthermore, the authors demonstrate that the standardization procedure used in conventional GRPO can harm calibration performance. They evaluate the calibration performance of PACE across various models and datasets and propose a confidence-based model routing strategy as a practical application.

Critical strengths
  • This paper presents a logically sound and well-structured argument. Specifically, the appropriate confidence at an intermediate reasoning step should represent the probability of eventually producing a correct answer given the current reasoning prefix (conditional expectation). These probabilities follow a consistent martingale structure as reasoning progresses (Doob martingale). Since these probabilities cannot be directly observed, the authors propose learning them using pre-generated correctness labels and the brier score (PACE).
  • The application of confidence estimates to model routing is also practical and valuable.
Critical weaknesses

1. Missing reference

  • The paper in [1] includes experiments investigating how the position at which confidence is elicited affects correctness probability. It would be beneficial to include this work as a reference.

2. Does the brier reward guarantee the desired conditional correctness probability?

If the population expected reward could be optimized exactly over all possible prefixes and continuations, the optimal confidence estimate would indeed correspond to the conditional probability of eventual answer correctness. This would require accounting for multiple possible correctness outcomes associated with the same prefix.

However, in practice, PACE relies on binary correctness labels observed from a finite set of frozen rollouts and expects the LLM to generalize to the probabilities of unseen continuations through shared model parameters.

Although the results in Figure 3-(f) provide indirect evidence that PACE captures trajectory-specific variations better than the baselines, the within-question variance measured at the AA position is only around 5%. This suggests that the learned confidence estimates may not be sufficiently differentiated to approximate the ideal conditional correctness probabilities.

To demonstrate that the brier reward actually leads to the desired conditional correctness probabilities at arbitrary positions tt, the authors should conduct an experiment in which multiple continuations are sampled from the same fixed prefix, and the resulting empirical success rate is compared against the confidence predicted by PACE.

[1] https://proceedings.mlr.press/v306/jang26a.html

Questions for authors

1. Comparison between PACE confidence and empirical accuracy estimated via monte carlo sampling

Could the authors provide results comparing PACE's predicted confidence with the empirical accuracy estimated through monte carlo sampling of multiple continuations from the same fixed prefix?

2. Within-question auroc analysis

Could the authors report the within-question auroc results? Since the auroc values in the main table are computed by pooling trajectories across different questions, a within-question auroc analysis would be necessary to evaluate whether PACE captures trajectory-specific correctness information more effectively than other methods, as intended by its design.

6 comments
AuthorsOpenReview ↗

response (1/2)

Thank you for the insightful and precise suggestions. We ran both analyses you proposed.

W2 / Q1: Does the Brier reward recover the conditional correctness probability?

Short answer: largely yes across questions, but only partly within a question. Against Monte Carlo estimates of VtV_t from 16 continuations per prefix, PACE's confidence correlates with VtV_t at r=0.51r=0.51–0.560.56 (Base-PACE-uu) and 0.450.45–0.580.58 (RLVR-PACE-uu) at every position, higher than every other readout, and it is calibrated against VtV_t (ECE 0.03–0.08). Within a question, however, it tracks VtV_t only weakly (rwithin≤0.18r_{\text{within}}\le0.18), as you suspected. The new Limitations section states this as the main open problem.

Details. Theorem 2 is indeed a population statement; whether a model trained on a finite frozen pool generalizes to VtV_t is an empirical question. We tested it with the Monte Carlo experiment you proposed:

  • Prefixes: each model's own DeepMath-500 solutions (500 questions × 4), cut at u∈{0.2,0.4,0.6,0.8}u\in\{0.2,0.4,0.6,0.8\}, plus each question's u=0u=0 state: 8,500 prefixes per model.
  • Target: from each prefix, 16 continuations sampled with the same model under the original prompt, temperature (0.6) and remaining token budget. V^t\hat V_t is the fraction of the 16 completed solutions that are correct, an unbiased estimate of VtV_t. As a check, at u=0u=0 it matches each model's pool pass rate to within 0.008.
  • Readouts: each model's confidence on its own prefixes, averaged over four samples.

Per-position values, as in Table 43 (each model on its own prefixes and scored against its own V^t\hat V_t). At each position uu:

  • rr is the Pearson correlation between the confidence ctc_t and V^t\hat V_t across all 2,000 prefixes, i.e. how well the confidence ranks prefixes by their success probability, across and within questions;
  • rwithinr_{\text{within}} is the same correlation after subtracting each question's mean (over its 4 prefixes) from both ctc_t and V^t\hat V_t. This removes question difficulty, so it measures whether the confidence distinguishes attempts at the same question. It is undefined at u=0u=0, where all of a question's prefixes are the same state.
Readoutrr @ 0.0rr @ 0.2rr @ 0.4rr @ 0.6rr @ 0.8ECE @ 0.0ECE @ 0.2ECE @ 0.4ECE @ 0.6ECE @ 0.8rwithinr_{\text{within}} @ 0.2rwithinr_{\text{within}} @ 0.4rwithinr_{\text{within}} @ 0.6rwithinr_{\text{within}} @ 0.8
Base-PACE-uu0.530.550.560.540.510.0780.0660.0660.0670.0710.040.120.090.09
PACE, u=0u{=}0 only0.500.450.450.440.440.0330.0320.0370.0530.0730.010.020.030.10
PACE, u=1u{=}1 only0.380.380.380.410.420.2470.2820.2830.2770.2540.010.010.050.10
Base-Verb−0.030.100.100.120.110.3480.3530.3530.3570.3600.05−0.01−0.020.04
RLVR-PACE-uu0.450.540.560.570.580.0390.0330.0260.0310.0400.090.090.180.18
RLVR-Verb0.020.090.090.090.060.1960.1960.1930.1960.2040.10−0.030.00−0.02
  • PACE tracks VtV_t at every position and is calibrated against it, not only against the final outcome. Training on uniformly sampled positions ranks prefixes better than training at either endpoint at every position, and RL for accuracy alone does not produce this: RLVR's own confidence barely correlates with VtV_t, while the same initialization after PACE reaches r=0.45r=0.45–0.580.58.
  • You are right that the trajectory-specific component is only partly recovered. PACE's rwithinr_{\text{within}} is positive at most positions but small, so its confidence mostly tracks question-level difficulty. This matches your reading of Figure 3(f), and the new Limitations section states it as the main open problem.

The revision adds this experiment to Section 6.2 and Appendix J.1 (Table 43).

AuthorsOpenReview ↗

response (2/2)

Q2: Within-question AUROC.

Short answer: PACE has the highest within-question AUROC at t=Tt=T (0.590 for Base-PACE-uu and 0.615 for RLVR-PACE-uu, narrowly above ConfTuner's 0.587 and CALIBER's 0.580), and it is the only method above chance at the midpoint (0.529 and 0.549, against 0.502).

Details. For each question with at least one correct and one incorrect solution in the method's own pool, we count the (correct, incorrect) solution pairs that the confidence orders correctly, pool the pairs within a dataset, and average over the seven datasets. Intervals come from a question-level bootstrap; "common" restricts every method to the 304 questions mixed in all pools, so that differences cannot come from which questions are compared.

Methodt≈T/2t\approx T/2t=Tt=T [95% CI]t=Tt=T, common
Base-Verb0.5020.526 [0.519, 0.533]0.531
RLVR-Verb0.5020.529 [0.520, 0.537]0.527
AvgProb—0.558 [0.543, 0.573]0.531
ProbeMLP—0.554 [0.537, 0.569]0.521
RLCR—0.542 [0.528, 0.556]0.543
RLCC-L1—0.533 [0.522, 0.545]0.537
CALIBER—0.580 [0.565, 0.594]0.563
ConfTuner—0.587 [0.571, 0.601]0.570
Base-PACE-uu0.5290.590 [0.578, 0.604]0.586
RLVR-PACE-uu0.5490.615 [0.596, 0.633]0.596
  • PACE ranks best at t=Tt=T, narrowly over ConfTuner and CALIBER (their intervals overlap with Base-PACE-uu's), and the ordering holds on the common questions.
  • At the midpoint only PACE is above chance. The untrained verbalizers stay at 0.502, and the other methods emit no midpoint confidence: coupled methods report only at their fixed slot, and AvgProb, ProbeMLP and ConfTuner are defined or fitted per endpoint.
  • Absolute values near 0.6 are expected. Telling apart two attempts at the same question is much harder than telling apart questions, for every method, consistent with the Monte Carlo result above.
  • At t=0t=0 every decoupled readout is at chance by construction, since all solutions of a question share the same probe state.

The revision adds this table as Appendix J.2 (Table 44), summarized in Section 6.2.

W1: Missing reference. Thank you. Jang et al. (ICML 2026) find that placing the confidence query before reasoning lowers accuracy after SFT, especially as KL regularization is removed, whereas after-answer queries do not. This independently supports Section 4 from a different angle: they vary the prompt position after training, while we isolate the causal effect of the announced value by forcing it (pass@1 moves by up to +0.21) and then remove the coupling by design. We now cite it in Section 2 and Section 4.

Thank you for your thoughtful responses.

Even though PACE shows highest Within-question AUROC, they still remain about 50% which means almost random guess.

I think this paper's hypothesis and experiments are quite interesting but considering the weak empirical results of their claims, I just maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response to all reviewers

We thank the reviewers for careful and constructive reviews. They found the target well motivated and the argument sound (ic7z), the problem setting novel and the Brier reward theoretically appealing (b85v), the design well-motivated and the evaluation careful (P3pm). Our original score is 3/2/3, after review, both reviewer ic7z and reviewer P3pm maintained their positive scores, while reviewer b85v did not join in the rebuttal process. The revised PDF marks every change in blue. The main additions:

  1. A direct Monte Carlo test of the target (ic7z). From each stored prefix we sample 16 continuations with the same model and compare PACE's confidence with their empirical success rate. PACE tracks it at every position (Section 6.2, Appendix J.1).
  2. Within-question AUROC (ic7z). PACE ranks solutions of the same question best at t=Tt=T and is the only method above chance at the midpoint (Section 6.2, Appendix J.2).
  3. Calibration at fixed token indices (P3pm). PACE never needs the final length; it stays calibrated through 1024 tokens and degrades only on the rare longest solutions (Section 5 "Deployment", Appendix J.3).
  4. The role of Section 4 and the attribution of BA confidence (b85v). The first paragraph of Section 4 explicitly explains its role to motivate the decoupled design. BA estimation is credited to Chen et al. (ICLR 2026), and Jang et al. (ICML 2026) is cited.
  5. A Limitations section (Section 7) that collects the caveats these analyses reveal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.