- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines confidence estimation for LLMs after agentic reinforcement learning changes their reasoning and tool-use trajectories. StaR-TTA trains a stability-to-reliability mapper and a trajectory-aware confidence scorer on labeled pre-RL trajectories, using hidden-state features, interaction statistics, and the consistency of five short answers sampled from a fixed trajectory prefix. A source-trained calibrator maps the scores to probabilities. For adaptation, the method fits a linear residual corrector on unlabeled post-RL examples, using rankings derived from answer stability and Base–RL answer agreement while keeping the scorer and calibrator frozen. The evaluation covers four QA benchmarks, transfer from TriviaQA to additional datasets, experiments in which estimators are retrained for other backbones, and both PPO and GRPO training.
Critical strengths
-
The problem is practically important: an agent may be updated through RL while its confidence estimator remains tied to an earlier model. Adapting confidence estimates without obtaining correctness labels for the updated agent is a useful deployment objective.
-
The evaluation spans several QA datasets, model families and scales, and two RL algorithms, and includes feature, loss, and adaptation ablations. This breadth represents a useful attempt to test whether the proposed training-and-adaptation recipe works beyond a single experimental setting.
Critical weaknesses
-
The evidence does not yet clearly establish the proposed mechanism or the necessity of its components. In Table 2, all three AUROC values for Tool control exactly match the RL values in Table 1 at the reported precision, although Section 3.2 describes replacing tool results. The reference values used for the parenthetical differences, the matched evaluation subsets, and the replay procedure are not specified in sufficient detail to assess the attribution to trajectory shift. The component results also require a more qualified interpretation: removing stability ranking on PopQA raises AUROC from 0.6956 to 0.7180 but worsens calibration (Table 12), while adding TTA worsens average ECE from 0.0654 to 0.0688 (Table 8). Although the existing ablations are useful, they do not isolate the hidden-state features or each stability statistic, and the loss ablation is reported only on PopQA. It therefore remains unclear when stability supervision helps discrimination, when it helps calibration, and when it imposes an incorrect ordering.
-
The incremental contribution over established consistency-based uncertainty estimation, along with the conditions required to reproduce and deploy the method, is not sufficiently delineated. Semantic entropy [1; 2], SelfCheckGPT [3], and the consistency hypothesis [4] are directly relevant. Although Table 13 provides useful stability-only baselines, these do not substitute for comparisons that isolate the proposed adaptation mechanism under the same sampling and feature budget. Reproducibility is also limited by the missing definitions of the 24 trajectory descriptors and the lack of detail on how the 64-dimensional hidden representation is constructed. The paper demonstrates QA transfer and retrains estimators across backbones, but does not establish a single task- and model-independent estimator. Finally, the reported 0.8% refers to extra sampled-answer tokens rather than end-to-end adaptation cost including Base-model execution. These omissions make it difficult to assess the method’s additional benefit, portability, and practical cost.
[1] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
[2] Detecting hallucinations in large language models using semantic entropy
[3] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
[4] The Consistency Hypothesis in Uncertainty Quantification for Large Language Models
Questions for authors
-
Why do the Tool control values in Table 2 exactly match the RL AUROCs in Table 1? Which reference values and evaluation subsets underlie the parenthetical differences?
-
How does StaR-TTA compare with semantic entropy under the same sampling budget, and with a scorer that uses the same features but neither of the SRM losses?
-
Why does removing stability ranking increase AUROC in Table 12? Does this effect persist across datasets and random seeds?
-
Why is test-time ranking based on an equally weighted average of stability and agreement signals rather than the learned SRM reliability?
-
What are the 24 trajectory descriptors, and how is the 64-dimensional representation constructed? What do h and each of the three stability statistics contribute independently?
-
Are estimators and correctors trained separately for each benchmark? Does “All” adaptation include the samples used in the final evaluation?
-
What are the total costs of adaptation on 64 questions, including Base-model execution and prefix processing, and how much additional latency does each subsequent prediction incur?
-
Does the method require a canonical short answer? How does it handle consistently wrong answers or situations in which RL correctly disagrees with the Base model?
6 comments
Response to Reviewer jmis 2
W1-e. The loss ablation is reported only on PopQA. W1-e. Expanded loss ablation: We extend the loss ablation to all four datasets. Since the stability-ranking loss is already analyzed in W1-c, we report the remaining three loss terms here. Overall, removing any loss term degrades performance, while SRM BCE shows the largest impact, particularly on HotpotQA, highlighting the importance of stability-to-reliability supervision.
| Dataset | Ablation Loss | AUROC ↑ | ECE ↓ | Brier ↓ | NLL ↓ | Avg. Drop (%) |
|---|---|---|---|---|---|---|
| TriviaQA | Ours | .800 | .054 | .181 | .542 | 0 |
| TriviaQA | Label BCE | .8017 | .0782 | .1849 | .5524 | 12.1473 |
| TriviaQA | Label Ranking | .8029 | .0573 | .1803 | .5414 | 1.3069 |
| TriviaQA | SRM BCE | .8039 | .0700 | .1816 | .5444 | 7.4673 |
| HotpotQA | Ours | .715 | .041 | .200 | .582 | 0 |
| HotpotQA | Label BCE | .7120 | .0404 | .2012 | .5862 | 0.0999 |
| HotpotQA | Label Ranking | .7139 | .0414 | .2004 | .5843 | 0.4508 |
| HotpotQA | SRM BCE | .6568 | .0770 | .2179 | .6293 | 28.2440 |
| MuSiQue | Ours | .672 | .120 | .145 | .532 | 0 |
| MuSiQue | Label BCE | .6248 | .1202 | .1466 | .5115 | 1.1165 |
| MuSiQue | Label Ranking | .6700 | .1218 | .1457 | .5390 | 0.8964 |
| MuSiQue | SRM BCE | .6654 | .1235 | .1464 | .5505 | 2.1010 |
W2-a. Expanded consistency-based baselines: We add Semantic Uncertainty, Semantic Entropy, SelfCheckGPT-NLI, and consistency-based aggregation under the same test-time sampling budget, while restricting all baselines to a feature dimensionality no larger than our 91-D input. StaR-TTA achieves the best AUROC on all four datasets and strong overall performance across ECE, Brier, and NLL. These results show that our gains go beyond sampling consistency alone and benefit from how stability is incorporated into confidence learning and test-time adaptation.
| Dataset | Method | AUROC ↑ | ECE ↓ | Brier ↓ | NLL ↓ |
|---|---|---|---|---|---|
| TriviaQA | Semantic Uncertainty (2023) | 0.7810 | 0.1471 | 0.2255 | 0.6432 |
| TriviaQA | Semantic Entropy (2024, DSE) | 0.6975 | 0.0884 | 0.2133 | 0.6174 |
| TriviaQA | SelfCheckGPT-NLI | 0.7301 | 0.1241 | 0.2185 | 0.6296 |
| TriviaQA | Consistency (Geom-agg) | 0.7358 | 0.0938 | 0.2064 | 0.6040 |
| TriviaQA | Ours | 0.8000 | 0.0540 | 0.1810 | 0.5420 |
| HotpotQA | Semantic Uncertainty (2023) | 0.6715 | 0.1326 | 0.2329 | 0.6643 |
| HotpotQA | Semantic Entropy (2024, DSE) | 0.6315 | 0.0895 | 0.2224 | 0.6343 |
| HotpotQA | SelfCheckGPT-NLI | 0.6439 | 0.1278 | 0.2304 | 0.6701 |
| HotpotQA | Consistency (Geom-agg) | 0.6377 | 0.0855 | 0.2200 | 0.6398 |
| HotpotQA | Ours | 0.7150 | 0.0410 | 0.2000 | 0.5820 |
| MuSiQue | Semantic Uncertainty (2023) | 0.6253 | 0.1057 | 0.1444 | 0.5032 |
| MuSiQue | Semantic Entropy (2024, DSE) | 0.6318 | 0.0879 | 0.1393 | 0.4718 |
| MuSiQue | SelfCheckGPT-NLI | 0.6470 | 0.1051 | 0.1427 | 0.5005 |
| MuSiQue | Consistency (Geom-agg) | 0.6245 | 0.0773 | 0.1356 | 0.4650 |
| MuSiQue | Ours | 0.6720 | 0.1200 | 0.1450 | 0.5320 |
| PopQA | Semantic Uncertainty (2023) | 0.6909 | 0.0647 | 0.2337 | 0.6593 |
| PopQA | Semantic Entropy (2024, DSE) | 0.6155 | 0.0227 | 0.2329 | 0.6572 |
| PopQA | SelfCheckGPT-NLI | 0.6373 | 0.0593 | 0.2349 | 0.6634 |
| PopQA | Consistency (Geom-agg) | 0.6381 | 0.0749 | 0.2347 | 0.6622 |
| PopQA | Ours | 0.6960 | 0.0610 | 0.2230 | 0.6350 |
W2-b-1 trajectory descriptors and Hidden representation construction: We will add detail explanation of each of the 24 trajectory descriptors and how the 64-dimensional representation constructed in our revision.
W2-b-2. Task- and model-agnostic design: Our framework is not tailored to any specific dataset or backbone. Table 4 demonstrates cross-dataset transfer, where an estimator trained on TriviaQA generalizes to unseen datasets, outperforming the strongest baseline by 8.5% AUROC. Tables 5 and 6 further show consistent improvements across different model families and model scales, with gains of 9.1% and over 8.0% AUROC, respectively. These results support the task- and model-agnostic nature of the proposed framework, while we do not claim that a single trained estimator transfers across all models without retraining.
Response to Reviewer jmis 3
W2-c & Q7. End-to-end adaptation cost: On average, StaR-TTA incurs approximately 4.1% additional token cost and 4.3% additional inference time relative to MICE-LR/RF, and 3.2% additional token cost and 3.3% additional inference time relative to HTC/CCPS.
| Dataset | Additional Generated Tokens (approx.) | vs. MICE-LR / MICE-RF | vs MICE-LR AUROC | vs. HTC / CCPS | vs CCPS AUROC |
|---|---|---|---|---|---|
| TriviaQA | 88,022 | 4.23% | 12.5% | 3.19% | 38.0% |
| HotpotQA | 82,703 | 3.71% | 7.5% | 3.00% | 27.7% |
| MuSiQue | 116,767 | 4.36% | 4.9% | 3.46% | 17.8% |
| PopQA | 61,034 | 4.09% | 10.1% | 3.14% | 27.1% |
| Avg. over four datasets | 87,132 | 4.10% | 8.75% | 3.20% | 27.65% |
Q2-a Add baseline: Please refer to our response to W2-a for details.
Q2-b. Scorer without SRM losses: We further construct a scorer using the same input features as our full method, but remove both SRM-related losses. Removing these losses substantially worsens ECE, Brier, and NLL across the datasets. The effect on AUROC is mixed, with slight improvements on TriviaQA and PopQA but decreases on HotpotQA and MuSiQue. These results indicate that SRM supervision primarily improves the reliability and calibration of the confidence scores, while its impact on ranking is more dataset-dependent.
| Dataset | Setting | AUROC ↑ | ECE ↓ | Brier ↓ | NLL ↓ | Avg. Degradation (%) ↓ |
|---|---|---|---|---|---|---|
| TriviaQA | Full method | 0.8000 | 0.0535 | 0.1808 | 0.5421 | 0.00 |
| TriviaQA | w/o both SRM losses | 0.8063 | 0.1075 | 0.1878 | 0.5612 | 26.89 |
| HotpotQA | Full method | 0.7154 | 0.0409 | 0.1996 | 0.5824 | 0.00 |
| HotpotQA | w/o both SRM losses | 0.7011 | 0.0836 | 0.2117 | 0.6106 | 29.33 |
| MuSiQue | Full method | 0.6721 | 0.1199 | 0.1451 | 0.5317 | 0.00 |
| MuSiQue | w/o both SRM losses | 0.6688 | 0.1405 | 0.1524 | 0.6252 | 10.07 |
| PopQA | Full method | 0.6956 | 0.0610 | 0.2226 | 0.6354 | 0.00 |
| PopQA | w/o both SRM losses | 0.7030 | 0.1168 | 0.2295 | 0.6560 | 24.19 |
Q3: We report the results of using 3 different random seed and find this effect persist.
| TTA seed | Full AUROC | w/o Stability Ranking |
|---|---|---|
| 20260918 | 0.694731 | 0.717380 |
| 20260919 | 0.696555 | 0.719384 |
| 20260920 | 0.695626 | 0.717099 |
Q4: SRM reliability is learned from the source model, while our test-time signal combines RL-side stability with Base–RL agreement to provide target-side information. Replacing it with SRM reliability reduces average AUROC from 0.7208 to 0.7025, while also worsening Brier and NLL, with nearly unchanged ECE.
| Dataset | TTA Correction Signal | AUROC ↑ | ECE ↓ | Brier ↓ | NLL ↓ |
|---|---|---|---|---|---|
| TriviaQA | Ours | 0.800 | 0.054 | 0.181 | 0.542 |
| TriviaQA | SRM | 0.800 | 0.054 | 0.181 | 0.543 |
| HotpotQA | Ours | 0.715 | 0.041 | 0.200 | 0.582 |
| HotpotQA | SRM | 0.709 | 0.033 | 0.201 | 0.586 |
| MuSiQue | Ours | 0.672 | 0.120 | 0.145 | 0.532 |
| MuSiQue | SRM | 0.610 | 0.127 | 0.150 | 0.568 |
| PopQA | Ours | 0.696 | 0.061 | 0.223 | 0.635 |
| PopQA | SRM | 0.691 | 0.061 | 0.224 | 0.638 |
Q5: Please refer to our response to W1-d for the results of removing each stability statistic features. We will add detail explanation of each of the 24 trajectory descriptors and how the 64-dimensional representation constructed in our revision.
Response to Reviewer jmis 1
We appreciate the reviewer’s constructive comments and provide our responses point by point below.
W1-a.&Q1 Correct Table 1 and Table2: We apologize for this data error and sincerely thank you for pointing it out. We have corrected the values in Table 1. In addition, we have revised the experimental settings in Table 2 for better reasonableness; please refer to our response to reviewer D78p W2‑a for details. The corrected tables are listed below for your reference.
Table 1
Table 2
W1-b. Parenthetical differences, the matched evaluation subsets, and the replay procedure: In the corrected table, we construct our analysis set by filtering trajectories with matching answers from the base and RL models, yielding 175 trajectories. Under tool control, the tool-retrieved content of RL is substituted with that of the base trajectory, and the parenthesized relative change is taken against base accuracy. Answer control uses samples with identical answers, and its relative change is also relative to the base trajectory. For trajectory replay, we feed the complete base trajectory — including both reasoning and search actions — into the same RL model to recompute confidence features. The relative change for trajectory replay is normalized against RL trajectory performance. The reported recovery measures the fraction of the RL-to-base performance gap that trajectory replay can recover.
W1-c. Stability ranking: We expand the ablation study to all four datasets as follows. Although removing stability ranking slightly improves AUROC on PopQA, it consistently worsens ECE, Brier, and NLL across all four datasets. This indicates that stability supervision consistently benefits calibration and probability quality, while its effect on confidence ranking can vary across datasets.
W1-d. Each stability statistic: We ablate each stability statistic individually on MuSiQue. Removing any of (S_{GA}), (S_{EC}), or (S_{FG}) worsens the calibration metrics, while removing all three produces the largest AUROC drop (0.6721→0.6578). Among the individual signals, (S_{GA}) contributes most to AUROC, whereas (S_{EC}) has the largest overall impact across metrics, suggesting complementary roles among the three stability signals.