acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

Abstract

Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: . This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose , a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of 0.721, with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
confidence estimation, agentic reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jmisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines confidence estimation for LLMs after agentic reinforcement learning changes their reasoning and tool-use trajectories. StaR-TTA trains a stability-to-reliability mapper and a trajectory-aware confidence scorer on labeled pre-RL trajectories, using hidden-state features, interaction statistics, and the consistency of five short answers sampled from a fixed trajectory prefix. A source-trained calibrator maps the scores to probabilities. For adaptation, the method fits a linear residual corrector on unlabeled post-RL examples, using rankings derived from answer stability and Base–RL answer agreement while keeping the scorer and calibrator frozen. The evaluation covers four QA benchmarks, transfer from TriviaQA to additional datasets, experiments in which estimators are retrained for other backbones, and both PPO and GRPO training.

Critical strengths
  1. The problem is practically important: an agent may be updated through RL while its confidence estimator remains tied to an earlier model. Adapting confidence estimates without obtaining correctness labels for the updated agent is a useful deployment objective.

  2. The evaluation spans several QA datasets, model families and scales, and two RL algorithms, and includes feature, loss, and adaptation ablations. This breadth represents a useful attempt to test whether the proposed training-and-adaptation recipe works beyond a single experimental setting.

Critical weaknesses
  1. The evidence does not yet clearly establish the proposed mechanism or the necessity of its components. In Table 2, all three AUROC values for Tool control exactly match the RL values in Table 1 at the reported precision, although Section 3.2 describes replacing tool results. The reference values used for the parenthetical differences, the matched evaluation subsets, and the replay procedure are not specified in sufficient detail to assess the attribution to trajectory shift. The component results also require a more qualified interpretation: removing stability ranking on PopQA raises AUROC from 0.6956 to 0.7180 but worsens calibration (Table 12), while adding TTA worsens average ECE from 0.0654 to 0.0688 (Table 8). Although the existing ablations are useful, they do not isolate the hidden-state features or each stability statistic, and the loss ablation is reported only on PopQA. It therefore remains unclear when stability supervision helps discrimination, when it helps calibration, and when it imposes an incorrect ordering.

  2. The incremental contribution over established consistency-based uncertainty estimation, along with the conditions required to reproduce and deploy the method, is not sufficiently delineated. Semantic entropy [1; 2], SelfCheckGPT [3], and the consistency hypothesis [4] are directly relevant. Although Table 13 provides useful stability-only baselines, these do not substitute for comparisons that isolate the proposed adaptation mechanism under the same sampling and feature budget. Reproducibility is also limited by the missing definitions of the 24 trajectory descriptors and the lack of detail on how the 64-dimensional hidden representation is constructed. The paper demonstrates QA transfer and retrains estimators across backbones, but does not establish a single task- and model-independent estimator. Finally, the reported 0.8% refers to extra sampled-answer tokens rather than end-to-end adaptation cost including Base-model execution. These omissions make it difficult to assess the method’s additional benefit, portability, and practical cost.

[1] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

[2] Detecting hallucinations in large language models using semantic entropy

[3] SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

[4] The Consistency Hypothesis in Uncertainty Quantification for Large Language Models

Questions for authors
  1. Why do the Tool control values in Table 2 exactly match the RL AUROCs in Table 1? Which reference values and evaluation subsets underlie the parenthetical differences?

  2. How does StaR-TTA compare with semantic entropy under the same sampling budget, and with a scorer that uses the same features but neither of the SRM losses?

  3. Why does removing stability ranking increase AUROC in Table 12? Does this effect persist across datasets and random seeds?

  4. Why is test-time ranking based on an equally weighted average of stability and agreement signals rather than the learned SRM reliability?

  5. What are the 24 trajectory descriptors, and how is the 64-dimensional representation constructed? What do h and each of the three stability statistics contribute independently?

  6. Are estimators and correctors trained separately for each benchmark? Does “All” adaptation include the samples used in the final evaluation?

  7. What are the total costs of adaptation on 64 questions, including Base-model execution and prefix processing, and how much additional latency does each subsequent prediction incur?

  8. Does the method require a canonical short answer? How does it handle consistently wrong answers or situations in which RL correctly disagrees with the Base model?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jmis 1

We appreciate the reviewer’s constructive comments and provide our responses point by point below.

W1-a.&Q1 Correct Table 1 and Table2: We apologize for this data error and sincerely thank you for pointing it out. We have corrected the values in Table 1. In addition, we have revised the experimental settings in Table 2 for better reasonableness; please refer to our response to reviewer D78p W2‑a for details. The corrected tables are listed below for your reference.

Table 1

MethodBase AUROC ↑RL AUROC ↑Relative drop (%)
MICE-LR0.7420.60518.46
MICE-RF0.7760.58225.04
CCPS0.7540.45539.67

Table 2

MethodBaseRL (natural)Tool controlAnswer controlTrajectory replayRecovery (%)
MICE-LR0.6310.485 (−.146)0.512 (−.119)0.512 (−.119)0.566 (+.054)55.5
MICE-RF0.6680.546 (−.122)0.583 (−.085)0.583 (−.085)0.627 (+.044)66.3
CCPS0.6580.481 (−.177)0.500 (−.158)0.500 (−.158)0.513 (+.013)18.1

W1-b. Parenthetical differences, the matched evaluation subsets, and the replay procedure: In the corrected table, we construct our analysis set by filtering trajectories with matching answers from the base and RL models, yielding 175 trajectories. Under tool control, the tool-retrieved content of RL is substituted with that of the base trajectory, and the parenthesized relative change is taken against base accuracy. Answer control uses samples with identical answers, and its relative change is also relative to the base trajectory. For trajectory replay, we feed the complete base trajectory — including both reasoning and search actions — into the same RL model to recompute confidence features. The relative change for trajectory replay is normalized against RL trajectory performance. The reported recovery measures the fraction of the RL-to-base performance gap that trajectory replay can recover.

W1-c. Stability ranking: We expand the ablation study to all four datasets as follows. Although removing stability ranking slightly improves AUROC on PopQA, it consistently worsens ECE, Brier, and NLL across all four datasets. This indicates that stability supervision consistently benefits calibration and probability quality, while its effect on confidence ranking can vary across datasets.

DatasetSettingAUROC↑ECE↓Brier↓NLL↓Avg degradation (%)↓
TriviaQAFull method0.80000.05350.18080.54210.00
TriviaQAw/o Stability Ranking0.80580.09890.18680.558722.63
HotpotQAFull method0.71540.04090.19960.58240.00
HotpotQAw/o Stability Ranking0.71030.08050.20940.603726.53
MuSiQueFull method0.67210.11990.14510.53170.00
MuSiQuew/o Stability Ranking0.67200.14290.15310.641620.67
PopQAFull method0.69560.06100.22260.63540.00
PopQAw/o Stability Ranking0.71800.11310.22500.645721.22

W1-d. Each stability statistic: We ablate each stability statistic individually on MuSiQue. Removing any of (S_{GA}), (S_{EC}), or (S_{FG}) worsens the calibration metrics, while removing all three produces the largest AUROC drop (0.6721→0.6578). Among the individual signals, (S_{GA}) contributes most to AUROC, whereas (S_{EC}) has the largest overall impact across metrics, suggesting complementary roles among the three stability signals.

DatasetSettingAUROC ↑ECE ↓Brier ↓NLL ↓Avg. Relative Degradation (%)
MuSiQueOurs0.67210.11980.14500.53160.0
MuSiQuew/o all cm0.65780.12410.14780.54221.99
MuSiQuew/o S_GA0.66580.12350.14650.54332.20
MuSiQuew/o S_EC0.67160.12490.14670.54742.97
MuSiQuew/o S_FG0.67190.12500.14670.54482.48
AuthorsOpenReview ↗

Response to Reviewer jmis 2

W1-e. The loss ablation is reported only on PopQA. W1-e. Expanded loss ablation: We extend the loss ablation to all four datasets. Since the stability-ranking loss is already analyzed in W1-c, we report the remaining three loss terms here. Overall, removing any loss term degrades performance, while SRM BCE shows the largest impact, particularly on HotpotQA, highlighting the importance of stability-to-reliability supervision.

DatasetAblation LossAUROC ↑ECE ↓Brier ↓NLL ↓Avg. Drop (%)
TriviaQAOurs.800.054.181.5420
TriviaQALabel BCE.8017.0782.1849.552412.1473
TriviaQALabel Ranking.8029.0573.1803.54141.3069
TriviaQASRM BCE.8039.0700.1816.54447.4673
HotpotQAOurs.715.041.200.5820
HotpotQALabel BCE.7120.0404.2012.58620.0999
HotpotQALabel Ranking.7139.0414.2004.58430.4508
HotpotQASRM BCE.6568.0770.2179.629328.2440
MuSiQueOurs.672.120.145.5320
MuSiQueLabel BCE.6248.1202.1466.51151.1165
MuSiQueLabel Ranking.6700.1218.1457.53900.8964
MuSiQueSRM BCE.6654.1235.1464.55052.1010

W2-a. Expanded consistency-based baselines: We add Semantic Uncertainty, Semantic Entropy, SelfCheckGPT-NLI, and consistency-based aggregation under the same test-time sampling budget, while restricting all baselines to a feature dimensionality no larger than our 91-D input. StaR-TTA achieves the best AUROC on all four datasets and strong overall performance across ECE, Brier, and NLL. These results show that our gains go beyond sampling consistency alone and benefit from how stability is incorporated into confidence learning and test-time adaptation.

DatasetMethodAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQASemantic Uncertainty (2023)0.78100.14710.22550.6432
TriviaQASemantic Entropy (2024, DSE)0.69750.08840.21330.6174
TriviaQASelfCheckGPT-NLI0.73010.12410.21850.6296
TriviaQAConsistency (Geom-agg)0.73580.09380.20640.6040
TriviaQAOurs0.80000.05400.18100.5420
HotpotQASemantic Uncertainty (2023)0.67150.13260.23290.6643
HotpotQASemantic Entropy (2024, DSE)0.63150.08950.22240.6343
HotpotQASelfCheckGPT-NLI0.64390.12780.23040.6701
HotpotQAConsistency (Geom-agg)0.63770.08550.22000.6398
HotpotQAOurs0.71500.04100.20000.5820
MuSiQueSemantic Uncertainty (2023)0.62530.10570.14440.5032
MuSiQueSemantic Entropy (2024, DSE)0.63180.08790.13930.4718
MuSiQueSelfCheckGPT-NLI0.64700.10510.14270.5005
MuSiQueConsistency (Geom-agg)0.62450.07730.13560.4650
MuSiQueOurs0.67200.12000.14500.5320
PopQASemantic Uncertainty (2023)0.69090.06470.23370.6593
PopQASemantic Entropy (2024, DSE)0.61550.02270.23290.6572
PopQASelfCheckGPT-NLI0.63730.05930.23490.6634
PopQAConsistency (Geom-agg)0.63810.07490.23470.6622
PopQAOurs0.69600.06100.22300.6350

W2-b-1 trajectory descriptors and Hidden representation construction: We will add detail explanation of each of the 24 trajectory descriptors and how the 64-dimensional representation constructed in our revision.

W2-b-2. Task- and model-agnostic design: Our framework is not tailored to any specific dataset or backbone. Table 4 demonstrates cross-dataset transfer, where an estimator trained on TriviaQA generalizes to unseen datasets, outperforming the strongest baseline by 8.5% AUROC. Tables 5 and 6 further show consistent improvements across different model families and model scales, with gains of 9.1% and over 8.0% AUROC, respectively. These results support the task- and model-agnostic nature of the proposed framework, while we do not claim that a single trained estimator transfers across all models without retraining.

AuthorsOpenReview ↗

Response to Reviewer jmis 3

W2-c & Q7. End-to-end adaptation cost: On average, StaR-TTA incurs approximately 4.1% additional token cost and 4.3% additional inference time relative to MICE-LR/RF, and 3.2% additional token cost and 3.3% additional inference time relative to HTC/CCPS.

DatasetAdditional Generated Tokens (approx.)vs. MICE-LR / MICE-RFvs MICE-LR AUROCvs. HTC / CCPSvs CCPS AUROC
TriviaQA88,0224.23%12.5%3.19%38.0%
HotpotQA82,7033.71%7.5%3.00%27.7%
MuSiQue116,7674.36%4.9%3.46%17.8%
PopQA61,0344.09%10.1%3.14%27.1%
Avg. over four datasets87,1324.10%8.75%3.20%27.65%

Q2-a Add baseline: Please refer to our response to W2-a for details.

Q2-b. Scorer without SRM losses: We further construct a scorer using the same input features as our full method, but remove both SRM-related losses. Removing these losses substantially worsens ECE, Brier, and NLL across the datasets. The effect on AUROC is mixed, with slight improvements on TriviaQA and PopQA but decreases on HotpotQA and MuSiQue. These results indicate that SRM supervision primarily improves the reliability and calibration of the confidence scores, while its impact on ranking is more dataset-dependent.

DatasetSettingAUROC ↑ECE ↓Brier ↓NLL ↓Avg. Degradation (%) ↓
TriviaQAFull method0.80000.05350.18080.54210.00
TriviaQAw/o both SRM losses0.80630.10750.18780.561226.89
HotpotQAFull method0.71540.04090.19960.58240.00
HotpotQAw/o both SRM losses0.70110.08360.21170.610629.33
MuSiQueFull method0.67210.11990.14510.53170.00
MuSiQuew/o both SRM losses0.66880.14050.15240.625210.07
PopQAFull method0.69560.06100.22260.63540.00
PopQAw/o both SRM losses0.70300.11680.22950.656024.19

Q3: We report the results of using 3 different random seed and find this effect persist.

TTA seedFull AUROCw/o Stability Ranking
202609180.6947310.717380
202609190.6965550.719384
202609200.6956260.717099

Q4: SRM reliability is learned from the source model, while our test-time signal combines RL-side stability with Base–RL agreement to provide target-side information. Replacing it with SRM reliability reduces average AUROC from 0.7208 to 0.7025, while also worsening Brier and NLL, with nearly unchanged ECE.

DatasetTTA Correction SignalAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQAOurs0.8000.0540.1810.542
TriviaQASRM0.8000.0540.1810.543
HotpotQAOurs0.7150.0410.2000.582
HotpotQASRM0.7090.0330.2010.586
MuSiQueOurs0.6720.1200.1450.532
MuSiQueSRM0.6100.1270.1500.568
PopQAOurs0.6960.0610.2230.635
PopQASRM0.6910.0610.2240.638

Q5: Please refer to our response to W1-d for the results of removing each stability statistic features. We will add detail explanation of each of the 24 trajectory descriptors and how the 64-dimensional representation constructed in our revision.

Reviewer_sVBNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the degradation of trajectory-based confidence estimation when transferring from a base language model to its Agentic RL counterpart. The authors identify reasoning-trajectory shifts as an important contributor and observe that answer stability under repeated sampling remains predictive of correctness after RL. Based on this observation, they propose StaR-TTA, which trains a confidence scorer using correctness and stability-derived supervision on base-model trajectories, then adapts a lightweight residual corrector using stability-based rankings from unlabeled target data. The language models, scorer, and calibrator remain frozen during adaptation. Experiments across four QA benchmarks achieve an average AUROC of 0.721, with additional evaluations covering different datasets, model families, scales, and RL algorithms. Ablations show that test-time adaptation improves average confidence ranking, although its effects on calibration vary across datasets.

Critical strengths
  1. The paper identifies substantial degradation in trajectory-based confidence estimation after Agentic RL and provides controlled experiments supporting trajectory shift as a contributing factor.
  2. StaR-TTA uses answer stability to adapt confidence scores without target correctness labels, updating only a lightweight residual corrector while keeping the language models and source estimator frozen.
Critical weaknesses
  1. Table 8 shows that TTA increases average AUROC from 0.7035 to 0.7208, but worsens average ECE from 0.0654 to 0.0688. On TriviaQA, all four metrics deteriorate: AUROC decreases from 0.8018 to 0.8000, ECE increases from 0.0455 to 0.0535, Brier score increases from 0.1794 to 0.1808, and NLL increases from 0.5386 to 0.5421. Thus, the evidence supports improved average ranking more strongly than consistently improved calibration.
  2. In Table 9, using only 2 or 4 adaptation samples reduces average AUROC to 0.6306 and 0.6680, respectively, compared with 0.7035 without adaptation. Although 64 samples recover performance to 0.7155, the absence of reported confidence intervals or repeated subset-sampling results makes the robustness of these gains difficult to assess.
Questions for authors
  1. Table 8 shows that TTA improves average AUROC from 0.7035 to 0.7208, but worsens average ECE from 0.0654 to 0.0688 and degrades all four metrics on TriviaQA. What explains these mixed results, and under what conditions should TTA improve both confidence ranking and calibration?
  2. Table 9 shows that using 2 or 4 adaptation samples reduces average AUROC to 0.6306 and 0.6680, compared with 0.7035 without adaptation. How robust are these results across random seeds and adaptation subsets, and can the authors report confidence intervals and provide guidance on selecting a sufficient adaptation sample size?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sVBN 1

We appreciate your constructive feedback. Our detailed responses to each point are provided below.

W1: Our TTA directly optimizes stability-guided confidence ordering rather than target calibration, explaining why AUROC gains are more pronounced while ECE may fluctuate slightly.

We would also like to clarify that our overall contribution is not limited to the final TTA correction step: StaR-TTA also includes stability-grounded source confidence learning, which establishes transferable reliability structure and substantially improves confidence estimation even before test-time correction.

Q1: On TriviaQA, when the adaptation signal conflicts with the original confidence ordering, it agrees with the ground-truth ordering in only 42.86% of cases, compared with 57.49% on MuSiQue. Thus, TTA is most effective when the adaptation signal provides reliable corrective information, but may offer limited or negative gains when the source estimator is already strong and the correction signal is noisy.

W2&Q2: Table 9 reports results averaged over three random seeds (20260918, 20260919, and 20260920), with the corresponding 95% confidence intervals. With very few adaptation samples, performance can be sensitive to subset sampling and may degrade. The results become relatively stable from 8 samples onward, while 64 samples recover most of the full-set gain.

NAverageTriviaQAHotpotQAMuSiQuePopQA
0.7035 ± .0000.8018 ± .0000.7112 ± .0000.6296 ± .0000.6714 ± .0000
1.7035 ± .0000.8018 ± .0000.7112 ± .0000.6296 ± .0000.6714 ± .0000
2.6306 ± .0392.7537 ± .1897.5125 ± .1821.6383 ± .0306.6181 ± .2580
4.6680 ± .1166.7828 ± .0516.5744 ± .3273.6671 ± .0310.6477 ± .1455
8.7089 ± .0020.7911 ± .0126.6839 ± .0220.6683 ± .0272.6924 ± .0170
12.7072 ± .0269.7914 ± .0151.6893 ± .0512.6703 ± .0325.6779 ± .0581
16.7056 ± .0377.7943 ± .0103.6811 ± .0855.6734 ± .0187.6738 ± .0591
32.7120 ± .0194.8002 ± .0030.7112 ± .0393.6709 ± .0225.6658 ± .0401
64.7155 ± .0028.8014 ± .0066.7151 ± .0119.6734 ± .0150.6723 ± .0088
All.7208 ± .0005.8000 ± .0008.7154 ± .0014.6721 ± .0015.6956 ± .0023
AuthorsOpenReview ↗

[Less Than 30 Hours Remaining] Could you confirm whether our responses have addressed your concern?

Dear Reviewer sVBN,

We sincerely appreciate your careful reading of our paper and the constructive feedback you have shared with us. Thank you for taking the time to examine our work closely and explain your concerns. We value the opportunity to learn from your perspective and to address the points you have raised through this discussion.

With the discussion deadline approaching on November 18 at 11:59 pm AoE, we wanted to kindly follow up and ask whether our explanations have addressed the issues you raised. We would be happy to clarify any remaining points while the discussion is still open, as we will be unable to answer further questions after the deadline.

Many thanks again for your time and thoughtful consideration.

Best regards, The authors

AuthorsOpenReview ↗

[Less Than 9 Hours Remaining] Could you let us know whether our replies have resolved your concerns?

Dear Reviewer sVBN,

As the author–reviewer discussion closes in less than 9 hours, we wanted to leave a short note to let you know that we remain available for any additional discussion you might find helpful.

If a particular explanation in our rebuttal would benefit from more detail, please feel free to point us to it. We would gladly revisit it and provide a response before November 18 at 11:59 pm AoE. We hope our replies have been useful, and we would sincerely appreciate any thoughts you may wish to share after reading them.

Thank you for making time for our submission among your other commitments. We value your input and the opportunity to make our work clearer through this exchange.

Best regards,

The authors

Reviewer_D78pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens to LLM confidence estimators when they are trained on a Base model's agentic (search-augmented) trajectories and then applied to the same model after agentic RL. Section 3 shows that trajectory-based estimators (MICE, CCPS, HTC) lose AUROC after Search-R1-style RL on NQ (Table 1). A controlled analysis (tool control, answer control, trajectory replay) first replaces RL tool results with Base ones, then restricts evaluation to examples where Base and RL give identical tool results and final answers, and finally replaces the RL reasoning trajectory with the Base one; it attributes part of the loss to the shift in reasoning trajectories (Table 2). The paper also argues that answer stability under resampling stays predictive of correctness after RL (Fig. 3). Stability is measured by fixing the trajectory prefix and sampling five final answers. The proposed method, StaR-TTA (Section 4), has three parts:

A source-trained Trajectory-Aware Confidence Scorer (TCS) over a 91-D feature vector: 64-D hidden state, 24-D trajectory statistics, and 3 stability features. Training with BCE and pairwise-ranking losses on correctness labels plus losses derived from a stability-to-reliability logistic mapper (SRM), followed by a Platt calibrator fitted on Base scores (it is frozen and not refitted after adaptation). A label-free test-time step that trains a zero-initialised linear residual corrector on unlabeled RL trajectories. Its targets are pairwise rankings from a score qttaq^{tta}, which averages RL stability features and Base–RL answer agreement (JSD), with an anchoring penalty that keeps scores near the source scores. The main experiments use Qwen2.5-3B with Search-R1 PPO on TriviaQA, HotpotQA, MuSiQue and PopQA, reporting AUROC, ECE, Brier and NLL against CCPS, MICE-LR/RF, HTC, P(Sufficient) and DACA. StaR-TTA reaches an average AUROC of 0.721 (Table 3). Further experiments cover cross-dataset transfer (train on TriviaQA, test on NQ, 2Wiki and Bamboogle; Table 4), Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct (Tables 5–6), GRPO (Table 7), and ablations of TTA, features, the agreement term and the losses, plus TTA sample size and cost (Tables 8–16).

Critical strengths
  1. A relevant, well-isolated problem with a useful diagnostic. Confidence estimators are trained on one model and then reused after that model is post-trained, so their failure under that shift is a real deployment issue that has received little attention. Table 1 shows large drops for trajectory-based estimators (CCPS −34.1%, MICE-RF −27.7% on NQ). Table 15 shows MICE-LR/RF and CCPS dropping on all four main datasets (e.g., CCPS falls from 0.787 to 0.420 AUROC on TriviaQA), although HTC does not drop on MuSiQue. The controlled analysis in Section 3.2 (tool control → answer control → trajectory replay, Table 2) tries to separate the cause rather than just report the drop. This framing and evidence is the paper's most durable contribution, and it is the main reason I see value in it.

  2. The proposed estimator is consistently stronger across the evaluated settings. The full method has the best AUROC on 3 of 4 datasets in the main setting (Table 3; HTC is better on MuSiQue, 0.721 vs 0.672). It has the best AUROC on all four datasets in cross-dataset transfer (Table 4), with Llama-3.2-3B (Table 5, +5.6 to +7.8 pts), with Qwen2.5-7B (Table 6, +1.1 to +10.2 pts) and with GRPO (Table 7, +3.4 to +9.2 pts), and usually the best Brier and NLL. I recomputed the stated averages and margins from the tables and they match (e.g., Table 3 averages 0.721/0.069/0.187/0.573). The extra sampling cost is small for these short-answer search-QA tasks (about 0.8% of tokens, Table 16, estimated on a small sample of Base trajectories; only the final answer is resampled, with the trajectory prefix fixed; running the Base model on the target questions during adaptation is extra, A.5). This consistency is what keeps the paper from a clear reject.

Critical weaknesses
  1. The gains are not shown to come from the titular contribution, label-free test-time adaptation (TTA); most come from a resampling-stability signal that the baselines do not get.

    • TTA adds only +1.7 AUROC on average (0.7035 → 0.7208, Table 8), and average ECE gets worse (0.0654 → 0.0688). Tables 4–7 have no w/o-TTA row, and no seeds or variances are reported anywhere.
    • Without TTA, the method already beats every baseline average in Table 3 by more than 5 pts. A 3-feature logistic regression on stability alone reaches 0.680 (Table 13), also above all baselines. No baseline is given the stability features.
    • The value of "label-free" is not quantified: there is no labeled reference (e.g., the same corrector trained on the same NN labeled RL examples), and 64 unlabeled questions already give most of the gain (Table 9).

    This matters because the title, abstract and method are framed around TTA, while the tables support a more modest claim: stability features help.

  2. The Section 3 motivation, which justifies an adaptation method, is thin and partly unexplained.

    • Table 2's "tool control" AUROCs (.567/.559/.491) are identical to the uncontrolled RL AUROCs in Table 1. The trajectory replay also recovers only 11–52% of the gap, and the rest is not explained.
    • "Stability is preserved after RL" (Fig. 3) is shown only in radar plots without numeric labels, so the text's claim that raw Sraw3S_{raw3} "consistently reaches around 0.7–0.8" AUROC cannot be checked. By my reading of the plots, HotpotQA and PopQA are nearer 0.66–0.68, so a table of the values would help.
Questions for authors
  1. [Clarification] For a fair attribution, could you (a) give the strongest baselines (e.g., MICE-RF and HTC) the same 3-D stability vector cmc^{m}, and (b) report a plain sampling-consistency baseline such as raw Sraw3S_{raw3}, or qttaq^{tta} used directly as the confidence score, per dataset on RL trajectories? If StaR-TTA's margin largely disappears under (a), my assessment of the contribution would change.
  2. [Clarification] Can you report w/ vs w/o TTA for the cross-dataset, Llama, 7B and GRPO settings (Tables 4–7)? Can you also give mean ± std over several seeds (TCS training and TTA subset sampling) for Table 8? This would show whether the +1.7-pt TTA effect is robust.
  3. [Clarification] Value of "label-free" and calibration. (a) The paper motivates label-free adaptation by the cost of target labels, but gives no labeled reference. Could you report the same linear corrector trained with BCE on the same NN labeled RL examples (e.g., N=64N=64, as in Table 9), and a Platt calibrator refitted on those labels? Please also support or soften the Appendix A.5 statement that other estimators may need retraining on tens of thousands of RL trajectories. Relatedly, the paper describes Search-R1 as trained with outcome-based rewards (Sec. 2), which normally require correctness labels on the RL training questions. Are the RL training questions and the evaluation questions from the same distribution, and in which deployment setting are target-model labels really unavailable? (b) The calibrator CψC_{\psi} is fitted on Base scores and never refitted (Sec. 4.3), which assumes the score-to-probability map is unchanged after RL. TTA raises average AUROC (0.7035 → 0.7208) but worsens average ECE (0.0654 → 0.0688) in Table 8; is the ECE increase consistent across Tables 4–7? (c) Does the 288288-D input ere^{r} of the corrector contain the three stability features? If so, does the corrector learn a shift correction or simply up-weight stability?
  4. [Clarification] Which unlabeled RL trajectories are used to fit the corrector in the "All" setting of Tables 3, 8 and 9? Are they the same test trajectories that are then scored, i.e., is the evaluation transductive? How were the NN-sample subsets in Table 9 drawn, and over how many draws? How were λa\lambda_a and the other hyperparameters selected: is "on training set" in A.3.1 the source split?
  5. [Clarification] Please explain the Table 2 "tool control" column, which exactly equals the Table 1 RL AUROCs for all three methods. Please also define the reference point for the deltas, give the size of the answer-controlled subset, and explain what still differs between Base and RL inputs under trajectory replay. Relatedly, why does MICE-LR score 0.567 on NQ in Table 1 but 0.653 on NQ in Table 4 after training on TriviaQA?
  6. [Clarification] Stability is measured by fixing the trajectory prefix and resampling only the final short answer (Sec. 3.3, 4.2, A.6). If the prefix already contains a wrong retrieval or a reasoning error, the final answers can be consistently wrong, giving high SGAS_{GA}/SECS_{EC}/SFGS_{FG} and an inflated confidence. The paper does not discuss this. Could you (a) stratify the stability-vs-correctness AUROC (Fig. 3 / Table 13) by whether the prefix contains the gold evidence, or by the number of invalid actions; (b) report, for Base and RL separately, the fraction of high-stability answers that are wrong, since RL often sharpens the output distribution and could raise stability without raising accuracy; and (c) compare prefix-fixed stability with stability from fully resampled trajectories on a subset? If high-stability errors increase after RL, the premise that the stability-correctness link is preserved (Sec. 3.3) would need qualification.
6 comments
AuthorsOpenReview ↗

Response to Reviewer D78p 1

Thank you for your constructive comments. We address each of your comments point by point below.

W1: (1) Identifying stability as a transferable reliability signal under RL-induced trajectory shift is one of our contributions. Using stability should therefore not be regarded merely as giving our method an additional signal unavailable to existing baselines. (2) We do not directly use stability as confidence. Instead, we integrate it with trajectory representations for instance-dependent test-time correction. (3) The gain of StaR-TTA should not be interpreted as only the gain from the final adaptation step. As shown in Table 13 and Table 3, the strongest stability-only/no-stability predictor achieves 0.6821/0.6465 AUROC, whereas our method reaches 0.7035 before the final adaptation and 0.7208 after adaptation, corresponding to gains of approximately +2.1 and +3.9 AUROC points, respectively.

W1-a. TTA results in Tables 4–7:We run experiments for the “without TTA” setting in Tables 4–7 with three different TTA seeds: 20260918, 20260919, and 20260920. We also report the 95% confidence interval for each metric. The average variance for the TTA setting is 0.000391, and the average variance for the w/o TTA setting is 0.007405. Avg across 4 datasets in each table:

Δ AUROC↑Δ ECE ↓Δ Brier ↓Δ NLL ↓
Table 4−0.023610−0.008442+0.001480+0.003649
Table 5−0.045583−0.001518+0.013823+0.038475
Table 6−0.033230−0.003987+0.012480+0.034280
Table 7−0.019227−0.007832+0.003018+0.006511

Table 4 results:

DatasetVersionAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQAw/o TTA0.79533±0.004610.09842±0.006780.19182±0.004050.57005±0.00903
TriviaQAwith TTA0.79952±0.000120.05484±0.000350.18231±0.000120.54592±0.00030
2WikiMultiHopQAw/o TTA0.58438±0.006170.07619±0.017550.24254±0.002990.67814±0.00656
2WikiMultiHopQAwith TTA0.62047±0.000420.08690±0.000240.24043±0.000180.67335±0.00045
Bambooglew/o TTA0.62869±0.024230.17322±0.021070.24519±0.007430.68326±0.01633
Bambooglewith TTA0.67655±0.001580.20393±0.001300.24576±0.000800.68735±0.00185
NQw/o TTA0.74175±0.002660.06347±0.017000.20428±0.001570.59508±0.00380
NQwith TTA0.74805±0.000150.09940±0.001050.20940±0.000170.60531±0.00039

Table 5 results:

DatasetVersionAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQAw/o TTA0.77928±0.001670.08340±0.012690.19729±0.002730.57997±0.00516
TriviaQAwith TTA0.83328±0.000050.04656±0.000290.16740±0.000010.50269±0.00000
HotpotQAw/o TTA0.66684±0.006800.04113±0.005630.19001±0.000340.56228±0.00117
HotpotQAwith TTA0.74579±0.000540.07391±0.000060.17932±0.000170.52760±0.00036
MuSiQuew/o TTA0.73151±0.008750.01429±0.002960.06851±0.000540.24877±0.00216
MuSiQuewith TTA0.74327±0.000230.04476±0.000070.07401±0.000030.26016±0.00008
PopQAw/o TTA0.79216±0.009960.11339±0.004020.19373±0.002520.56976±0.00533
PopQAwith TTA0.82978±0.000640.09305±0.000250.17351±0.000440.51643±0.00110

Table 6 results:

DatasetVersionAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQAw/o TTA0.79163±0.002540.11593±0.001250.19997±0.000440.58261±0.00110
TriviaQAwith TTA0.82225±0.000380.11209±0.000220.17977±0.000190.53937±0.00052
HotpotQAw/o TTA0.74684±0.004020.07226±0.004290.20290±0.000920.59227±0.00180
HotpotQAwith TTA0.76918±0.000500.08102±0.000210.19710±0.000330.56930±0.00074
MuSiQuew/o TTA0.75604±0.012080.08442±0.010310.14314±0.003200.44892±0.01342
MuSiQuewith TTA0.80521±0.000120.03783±0.000320.12066±0.000020.38161±0.00005
PopQAw/o TTA0.76848±0.007020.04550±0.004730.19471±0.002240.57092±0.00426
PopQAwith TTA0.79928±0.000110.10312±0.000510.19326±0.000060.56731±0.00008
AuthorsOpenReview ↗

Response to Reviewer D78p 2

Table 7 results:

DatasetVersionAUROC ↑ECE ↓Brier ↓NLL ↓
TriviaQAw/o TTA0.74984±0.002420.05080±0.016520.20107±0.003000.58651±0.00761
TriviaQAwith TTA0.75836±0.000140.05146±0.000600.19916±0.000060.58533±0.00017
HotpotQAw/o TTA0.69134±0.015830.10620±0.002900.17872±0.000920.58162±0.02168
HotpotQAwith TTA0.72357±0.000020.11747±0.000210.19197±0.000140.56653±0.00027
MuSiQuew/o TTA0.66204±0.008330.04589±0.011030.04988±0.001160.20472±0.00525
MuSiQuewith TTA0.67627±0.000280.04783±0.000060.04751±0.000030.19906±0.00005
PopQAw/o TTA0.72628±0.013670.06978±0.016140.20591±0.004350.60097±0.00950
PopQAwith TTA0.75022±0.000150.09653±0.000240.20403±0.000160.59602±0.00033

W1-b. Stability-enhanced baseline: We would like to clarify that using stability signals is itself one of our contributions. We augment MICE-RF by concatenating our 3-D stability vector with its original features and retraining it under the same source-training protocol. Stability improves MICE-RF from 0.6111 to 0.6689 AUROC, while StaR-TTA further reaches 0.7208. This indicates that our advantage does not come from the stability features alone, but from how they are incorporated into the confidence estimation and adaptation pipeline.

MethodTriviaQAHotpotQAMuSiQuePopQAAVG
MICE-RF.6290.6149.6101.5903.6111
MICE-RF + Stability.7512.6689.6158.6399.6689
Ours.8000 (+.0488).7154 (+.0465).6721 (+.0562).6956 (+.0557).7208 (+.0518)

W1-c. Quantified value of our "label-free" setting:We quantify the value of the “label-free” setting by retraining the existing baseline MICE-LR on RL trajectories with 256, 512, 1024, 2048, and 24669 labeled samples, for comparison against our label-free approach. The results are provided below.

MethodTotal LablesAUROC ↑ECE ↓Brier ↓NLL ↓
MICE-LR2560.61070.14700.23230.7018
MICE-LR5120.62770.13080.22380.6721
MICE-LR1,0240.64640.09950.21080.6249
MICE-LR2,0480.65930.07080.20070.5926
MICE-LR24,6690.70020.04000.18720.5531
HTC2560.65390.13180.22350.7943
HTC5120.68260.08090.20120.6210
HTC1,0240.69630.06700.19420.5804
HTC2,0480.71270.04930.18650.5542
HTC24,6690.73760.03440.17860.5350
Ours00.72080.06880.18700.5729

W2-a. Correction and clarification of Table 1/2. We apologize for the mistake: the RL AUROCs in Table 1 were incorrectly filled. The corrected Table 1 is shown below.

MethodBase AUROC ↑RL AUROC ↑Relative drop (%)
MICE-LR0.7420.60518.46
MICE-RF0.7760.58225.04
CCPS0.7540.45539.67

In the original Table 2, tool control was evaluated on 500 test samples, while answer control and trajectory replay used only the subset with matching base and RL answers. For fair comparison, we rerun all settings on this consistent subset and update Table 2. The new results are shown below.

MethodBaseRL (natural)Tool controlAnswer controlTrajectory replayRecovery (%)
MICE-LR0.6310.485 (−.146)0.512 (−.119)0.512 (−.119)0.566 (+.054)55.5
MICE-RF0.6680.546 (−.122)0.583 (−.085)0.583 (−.085)0.627 (+.044)66.3
CCPS0.6580.481 (−.177)0.500 (−.158)0.500 (−.158)0.513 (+.013)18.1
AuthorsOpenReview ↗

Response to Reviewer D78p 3

W2-b. Explain the rest of the drop: Trajectory replay controls the observable reasoning trajectory, but it does not revert the RL-updated model parameters or the resulting internal representation shift. Such internal changes are difficult to isolate through the same controlled intervention, and may account for part of the remaining gap.

W2-c. Detailed numerical values for Figure 3:Thank you for pointing this out. We provide the complete numerical values below and have revised the corresponding statement in the revised manuscript.

DatasetModelSGASECSFGSraw3
TriviaQABase0.72770.70660.68780.7186
TriviaQAAgentic-RL0.74840.71620.70910.7446
HotpotQABase0.68120.64040.61970.6621
HotpotQAAgentic-RL0.67610.65400.64620.6771
MuSiQueBase0.70330.70300.66290.7021
MuSiQueAgentic-RL0.65140.64340.62900.6539
PopQABase0.67790.65180.64320.6644
PopQAAgentic-RL0.66670.64120.63920.6642

Q1-a. Stability features enhanced baseline: Please refer to our response to W1-c. for details.

Q1-b plain sampling-consistency baseline: Directly using (S_{\mathrm{raw3}}) as the confidence score achieves 0.6850 AUROC on average, while StaR-TTA further improves this to 0.7208 (+3.6 pts), showing that the improvement comes not only from the stability signal itself, but also from how it is incorporated into our framework.

DatasetSraw3ours
TriviaQA0.74460.800
HotpotQA0.67710.715
MuSiQue0.65390.672
PopQA0.66420.696

Q2-a. w/o TTA for Table 4-7: Please refer to our respones to W1-a. for details.

Q2-b. std for Table 8:The results in Table 8 are obtained with the same three seeds (20260918, 20260919, and 20260920), with an average standard deviation of 0.000406 reported.

Q3-a-1. Retrain the baseline with RL label: Please refer to our response to W1‑c for details.

Q3-a-2. Retrain Platt with RL label: we retrain Platt on labeled RL trajectories with 24669 labeled samples, for comparison against our label-free approach. The results are provided below.

Labeled SamplesPlattours (no label)
246990.69540.7208 (no label)

Q3-a-3. Other estimators may need retraining on tens of thousands of RL trajectories: Our experiment on MICE-LR shows that it needs training on 24699 samples before it can approach the performance of our method. Please refer to our response to W1-c.

Q3-a-4. Distribution: The RL problems and evaluation problems are not drawn from the same distribution. In fact, we follow the setting of Search‑R1: for example, we can train on the NQ and Hotpot dataset and test on other dataset.

Q3-a-5. Target-model labels unavailable deployment setting: In unseen domains, target correctness labels are often unavailable because obtaining them usually requires additional manual annotation. Our setting reflects this unlabeled deployment scenario.

Q3-b-1. w/o TTA for Table 4-7: Please refer to our response to W1-a for the full results. Across Tables 4–7, TTA consistently improves AUROC, Brier, and NLL, while ECE shows a small increase. This suggests that TTA is particularly effective in improving confidence ranking, while also benefiting overall probabilistic prediction quality.

Δ AUROC↑Δ ECE↓Δ Brier↓Δ NLL ↓
Table 4−0.023610−0.008442+0.001480+0.003649
Table 5−0.045583−0.001518+0.013823+0.038475
Table 6−0.033230−0.003987+0.012480+0.034280
Table 7−0.019227−0.007832+0.003018+0.006511

Q3-b-2. Shift correction : Yes, it contains three stability features. We replace the corrector with a weight-only scaling module, which learns a non-negative coefficient and adds this coefficient to the output. The results below demonstrate that simply learning scaling weights yields worse performance than our current corrector, indicating that the corrector does not merely learn to upweight values.

DatasetWeight-only stability scalingOurs (full TTA)
TriviaQA0.79890.8000
HotpotQA0.70910.7154
MuSiQue0.65190.6721
PopQA0.67000.6956
AVG0.70750.7208

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.