acceptodds
Under review as a conference paper at ICLR 2027

TTCL: Test-time Calibration Learning for Large Language Model Reasoning

Abstract

Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incorporate calibration learning into reinforcement learning (RL), jointly optimizing answer correctness and verbalized confidence using ground-truth correctness supervision. However, their reliance on labeled data limits their applicability in practical test-time settings, where ground-truth labels are unavailable and calibration may need to adapt to newly encountered target tasks. To address this challenge, we propose Test-Time Calibration Learning (TTCL), a label-free framework that jointly adapts reasoning accuracy and verbalized confidence directly on unlabeled target-task data. Specifically, TTCL derives self-supervision signals for both correctness and calibration from multiple model-generated responses, enabling calibration learning at test time without ground-truth labels. Theoretical analysis further establishes TTCL as a bounded surrogate for the ideal calibration objective. Extensive experiments on mathematical reasoning and factual question answering demonstrate that TTCL consistently improves both accuracy and calibration across diverse models and tasks. On base models, TTCL achieves an average relative accuracy improvement of and an ECE reduction of across eight benchmarks. Moreover, TTCL can further improve both accuracy and calibration for already calibrated models under domain shift, particularly when source-domain calibration transfers poorly to target tasks. In the math-to-FactQA setting, TTCL achieves an average relative accuracy gain of and reduces ECE by .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Calibration, Uncertainty Estimation, Reinforcement Learning, Large Language Model Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XxFaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TTCL adapts LLM reasoning and confidence on unlabeled target tasks. It derives rewards from sampled-answer agreement and frequencies, with evaluation on mathematical reasoning and factual QA.

Critical strengths

Label-free calibration under domain shift is a useful problem. Separating answer and confidence rewards is a sensible design choice.

Critical weaknesses
  1. The calibration results contain a substantial inconsistency. For Qwen3-4B with TTCL on NQ, Tables 2 and 11 report accuracy 0.1063, mean confidence 0.420, and ECE 0.155. Under Eq. (3), ECE must be at least |0.420 − 0.1063| = 0.3137 if these use the same predictions.

  2. The surrogate guarantee depends on an unquantified gap between answer frequency and correctness. Frequent but incorrect answers remain a fundamental concern.

  3. The benefit of test-time RL over directly using answer frequencies as confidence scores is unclear, especially given the cost of 10–40 training epochs.

Questions for authors
  1. Do Tables 2 and 11 use the same predictions and checkpoints? Please reconcile the reported metrics.

  2. Can you quantify the frequency–correctness gap and evaluate frequent incorrect answers with the target cache enabled?

  3. Could you compare TTCL with direct frequency-based confidence under matched compute budgets and report the adaptation time?

12 comments
AuthorsOpenReview ↗

Response to reviewer XxFa (Part 1/4)

W1: The calibration results contain a substantial inconsistency. For Qwen3-4B with TTCL on NQ, Tables 2 and 11 report accuracy 0.1063, mean confidence 0.420, and ECE 0.155. Under Eq. (3), ECE must be at least |0.420 − 0.1063| = 0.3137 if these use the same predictions.
Q1: Do Tables 2 and 11 use the same predictions and checkpoints? Please reconcile the reported metrics.

A: Thanks for your valuable comments. We agree that you are correct that the originally reported values cannot describe the same set of predictions. We would like to clarify that the inconsistency arose when we updated the tables using results from a later checkpoint. Specifically, we updated ACC, ECE, Brier score, and AUROC using the step-60 checkpoint, but inadvertently retained the mean confidence of 0.420 from the step-40 checkpoint. The corresponding mean confidence for the step-60 checkpoint is 0.221. We apologize for this reporting error. For transparency, we provide the evaluation results from both checkpoints below:

CheckpointACCMean ConfECEBrierAUROC
Step 4010.130.4200.3490.2340.755
Step 6010.630.2210.1550.1760.848

Thus, the original entries in Tables 2 and 11 inadvertently combined metrics from different checkpoints. The corrected entry consistently reports the step-60 results, replacing 0.420 with 0.221 in Table 11 while retaining the ACC, ECE, Brier score, and AUROC listed above.

W2: The surrogate guarantee depends on an unquantified gap between answer frequency and correctness. Frequent but incorrect answers remain a fundamental concern.
Q2: Can you quantify the frequency–correctness gap and evaluate frequent incorrect answers with the target cache enabled?

A: Thanks for your contrustive comments. We agree that quantifying the discrepancy between frequency-based supervision and correctness is important for interpreting the surrogate bound. Theorem 4.1 connects the TTCL objective to the ideal calibration objective through the frequency-correctness estimation error. Following your suggestions, to provide a quantitative analysis, we introduce the Target-Correctness Gap, which measures the mean absolute difference between each sampled answer’s frequency and its observed correctness:

Target-Correctness Gap=1NG∑i=1N∑j=1G∣q(aij∣xi)−zij∣,\mathrm{Target\text{-}Correctness\ Gap} = \frac{1}{NG} \sum_{i=1}^{N}\sum_{j=1}^{G} \left| q(a_{ij}\mid x_i)-z_{ij} \right|,

where NN is the number of questions, GG is the number of responses sampled per question (GG=32 here), q(aij∣xi)q(a_{ij}\mid x_i) is the cached frequency of answer aija_{ij}, and zij∈{0,1}z_{ij}\in\{0,1\} indicates whether that answer is correct. This metric ranges from 0 to 1, with smaller values indicating closer agreement between answer frequency and observed correctness. Ground-truth answers are used only for this post-hoc analysis and are not accessed during TTCL training. We examine this metric across multiple models and benchmarks, with the results reported below:

Target-Correctness GapMATH500AMCAIME24AIME25DAPO-MathLightEval-MATHNQHotpotQATriviaQA
Qwen3-4B0.11980.17380.18620.19130.14710.10100.19310.33590.3257
Qwen3-8B0.15730.15750.18030.19110.21020.20750.21570.30880.2948
Llama-3.1-8B-Instruct0.19920.18480.07680.05570.15910.17940.25680.26600.2769

From these results, we observe that the values of Target–Correctness Gap remain within a relatively small range. This metric characterizes the discrepancy between frequency-based self-supervision targets and observed correctness, providing an empirical diagnostic related to the estimation error in Theorem 4.1. These results suggest that answer frequency can serve as a useful proxy for latent correctness. Together with our theoretical analysis, they indicate that TTCL provides a reasonable surrogate for calibration learning in the absence of ground-truth labels.

Complementing this analysis, the training dynamics in Figures 5, 11, and 12 in the manuscript show that TTCL progressively narrows the gap between mean confidence and empirical accuracy. This provides evidence that optimizing the TTCL objective improves the alignment between the model’s expressed confidence and its actual accuracy. Taken together, the quantified frequency-correctness gap, theoretical analysis, extensive experiments, and observed training dynamics support both the rationale for TTCL’s calibration objective and its effectiveness in improving confidence estimation of the model. We will update these results and analyses into the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer XxFa (Part 2/4)

W3: The benefit of test-time RL over directly using answer frequencies as confidence scores is unclear, especially given the cost of 10-40 training epochs.
Q3: Could you compare TTCL with direct frequency-based confidence under matched compute budgets and report the adaptation time?

A: Thanks for your valuable comments. We would like to clarify that our motivation is to enable the model to learn to express its own confidence without ground-truth annotations. Through TTCL, the model learns to directly express how likely its answer is to be correct, incorporating this confidence estimation capability into its parameters. Although TTCL requires multiple rollouts for a question, these are used during adaptation to train the model’s capability of confidence expression. After adaptation, the model can produce an answer and its verbalized confidence in a single inference pass.

Following your suggestions, we futher compare TTCL with the frequency-based confidence estimation, referred as SampleConsistency, which samples multiple rollouts and uses the majority answer as the prediction and frequency as the confidence. We use the number of multiple rollouts K=32K=32, matching the rollout group size used during TTCL adaptation. We report Pass@1, Avg@32, Maj@32 and ECE, where Pass@1 and Avg@32 assess the sampling efficiency, Maj@32 examines whether learning from frequency-based supervision improves majority-vote accuracy over directly using the initial model’s sampling results, while ECE measures calibration performance. The results are reported below:

ModelMethodMATH500AMCAIME24AIME25
Pass@1 ↑Avg@32 ↑Maj@32 ↑ECE ↓Pass@1 ↑Avg@32 ↑Maj@32 ↑ECE ↓Pass@1 ↑Avg@32 ↑Maj@32 ↑ECE ↓Pass@1 ↑Avg@32 ↑Maj@32 ↑ECE ↓
Qwen3-4BSampleConsistency79.0081.3487.600.04162.5065.0880.000.08320.0019.5826.670.11913.3317.5023.330.147
TTCL87.4086.8490.000.02280.0078.9882.500.08930.0028.2330.000.06823.3321.3523.330.245
Qwen3-8BSampleConsistency77.6077.7887.800.08467.5065.3180.000.07520.0024.5836.670.05720.0018.4423.330.111
TTCL88.8088.6690.200.08277.5079.3885.000.10326.6729.5840.000.07326.6726.1526.670.109
Llama-3.1-8B-InstructSampleConsistency42.2040.6656.600.06215.0019.5327.500.0780.003.546.670.1570.000.000.000.096
TTCL56.4055.3862.600.10142.5043.7550.000.10910.009.4810.000.2160.000.000.000.114

From these results, we observe that TTCL generally improves Pass@1 and Avg@32, indicating better sampling efficiency. Its improved Maj@32 further shows that learning from frequency-based supervision can enhance the model’s answer distribution beyond the initial model’s voting performance. Even in some cases, TTCL’s single-response accuracy even exceeds the initial model’s majority-vote accuracy. For calibration, TTCL achieves lower or comparable ECE to the initial model’s frequency-based confidence estimates. These results suggest that TTCL adaptation enables the model to internalize confidence estimation from multiple rollouts and express it directly through verbalized confidence, allowing the model to assess the correctness of its answer in a single response without repeated sampling at inference time. This reduces the number of sampled responses per question from K to one, yielding an approximately K-fold reduction in inference-time sampling cost.

[This response continues in the next part.]

AuthorsOpenReview ↗

Response to reviewer XxFa (Part 3/4)

[continued response to W3 and Q3]

Additionally, to examine whether this learned capability generalizes to unseen questions, we supplement the experiments with a held-out setting, where the target-task data are divided into an unlabeled adaptation set and a disjoint test set. TTCL is trained only on the adaptation set and evaluated on the test set. Specifically, for mathematical reasoning, we randomly sample 20 examples from each of MATH500, AMC, AIME24, and AIME25 to form an unlabeled adaptation set of 80 examples for TTCL training, and use the remaining examples from each benchmark for evaluation; for factual QA, we randomly sample 100 examples from each of SimpleQA, NQ, HotpotQA, and TriviaQA as the unlabeled adaptation set of 400 examples for TTCL training, and use the remaining examples for evaluation. The results are reported below:

ModelMethodMATH500AMCAIME24AIME25
ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑
Qwen3-4BBase Model80.210.1810.65967.500.3180.69218.750.7880.73318.750.7220.697
TTCL83.540.0990.88368.750.0750.81822.500.1850.85023.750.2680.939
#Improv+4.16%+45.48%+34.09%+1.85%+76.51%+18.25%+20.00%+76.52%+15.91%+26.67%+62.83%+34.66%
Qwen3-8BBase Model80.210.1520.59866.250.3100.57321.250.7740.51325.000.7410.619
TTCL83.130.0600.84570.630.1130.89325.000.2800.92327.500.2690.814
#Improv+3.64%+60.70%+41.24%+6.60%+63.49%+55.92%+17.65%+63.78%+79.85%+10.00%+63.66%+31.60%
Llama-3.1-8B-InstructBase Model39.580.4330.53120.630.6140.4462.500.8090.2710.000.865-
TTCL41.880.1770.71233.130.1540.6167.500.3270.7140.000.391-
#Improv+5.79%+59.12%+34.09%+60.59%+74.92%+38.12%+200%+59.58%+163.5%-+54.80%-
RLCR-calibrated Qwen3-4B on FactQA-10KBase Model80.210.1810.65967.500.3180.69218.750.7880.73318.750.7220.697
RLCR Model81.670.1170.76171.250.1430.65321.250.2670.81023.750.3310.839
TTCL82.290.0860.90575.000.1140.90622.500.1740.91426.250.1680.931
#Improv+0.76%+26.50%+18.92%+5.26%+20.28%+38.74%+5.88%+34.83%+12.84%+10.53%+49.24%+10.97%
ModelMethodSimpleQANQHotpotQATriviaQA
ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑
Qwen3-4BBase Model1.670.9070.4748.000.8630.7567.880.8240.66518.750.7620.674
TTCL2.000.2500.72910.500.2750.8368.500.2410.71920.380.2480.837
#Improv+19.76%+72.44%+53.80%+31.25%+68.13%+10.58%+7.87%+70.75%+8.12%+8.69%+67.45%+24.18%
RLCR-calibrated Qwen3-4B on DAPO-Math-14KBase Model1.670.9070.4748.000.8630.7567.880.8240.66518.750.7620.674
RLCR Model2.000.6100.65710.880.6590.7568.500.5610.80222.380.5280.729
TTCL2.670.3280.72513.630.3380.8249.130.3290.86923.380.2980.856
#Improv+33.50%+46.23%+10.35%+25.28%+48.71%+8.99%+7.41%+41.35%+8.35%+4.47%+43.56%+17.42%
RLCR-calibrated Qwen3-8B on DAPO-Math-14KBase Model2.330.8920.50910.250.8680.6268.380.8500.65826.500.7040.558
RLCR Model2.440.7590.61713.250.7310.6609.750.6190.66930.750.5540.708
TTCL2.780.5060.68015.250.4590.79010.620.4760.69232.380.3800.756
#Improv+13.93%+33.33%+10.21%+15.09%+37.21%+19.70%+8.92%+23.10%+3.44%+5.30%+31.41%+6.78%

From these results, we observe that TTCL generally improves accuracy while consistently reducing ECE and improving AUROC on the held-out test sets. These gains are observed for both the initial models and the RLCR-calibrated models, indicating that TTCL remains effective when adaptation and evaluation examples are disjoint. In particular, TTCL further improves models calibrated on a different source domain, including FactQA-calibrated models evaluated on mathematical reasoning and math-calibrated models evaluated on factual QA. Together, these results suggest that the confidence estimation capability learned through TTCL generalizes to unseen target-task examples.

[This response continues in the next part.]

Reviewer_i6LqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTCL, a framework for jointly improving answer accuracy and verbalized confidence on unlabeled target-task data. It uses majority-vote pseudo-labels for correctness and answer frequencies as confidence targets, with separate reward application and a temporally lagged target cache. Experiments on mathematical reasoning and factual QA report improved accuracy and calibration.

Critical strengths
  • TTCL addresses the practical problem of calibration without target-task labels. Extensive experiments across multiple models and benchmarks, including domain-shift settings, demonstrate its effectiveness.
Critical weaknesses
  • Insufficient validation of the core supervision assumption. Frequent answers can be incorrect, while correct answers may appear infrequently, particularly on challenging questions. TTCL may therefore reinforce incorrect answers while encouraging high confidence in them. The target cache slows feedback but does not correct erroneous supervision. This suggests a potential limitation when answer frequency provides an unreliable signal of correctness/confidence.
  • Limited methodological novelty. Majority-vote pseudo-labeling, consistency-based confidence estimation, and confidence distillation have been explored in prior work. The contribution appears to lie mainly in combining these components for test-time adaptation.
Questions for authors
  • How does TTCL affect pass@k and maj@k as k increases? Since the method reinforces dominant answers, it may reduce sampling diversity and suppress infrequent correct responses, potentially limiting the benefits of additional test-time sampling.
11 comments
AuthorsOpenReview ↗

Response to reviewer i6Lq (Part 1/4)

W1: Insufficient validation of the core supervision assumption. Frequent answers can be incorrect, while correct answers may appear infrequently, particularly on challenging questions. TTCL may therefore reinforce incorrect answers while encouraging high confidence in them. The target cache slows feedback but does not correct erroneous supervision. This suggests a potential limitation when answer frequency provides an unreliable signal of correctness/confidence.

A: Thanks for your valuable comments. We agree that frequent answers can be incorrect and that the target cache stabilizes supervision rather than correcting erroneous pseudo-labels. Following your suggestions, to examine the reliability of this supervision signal, we quantify how closely the frequency-based targets agree with answer correctness. Specifically, we report the Target-Correctness Gap, defined as:

Target-Correctness Gap=1NG∑i=1N∑j=1G∣q(aij∣xi)−zij∣,\mathrm{Target\text{-}Correctness\ Gap} = \frac{1}{NG} \sum_{i=1}^{N}\sum_{j=1}^{G} \left| q(a_{ij}\mid x_i)-z_{ij} \right|,

Here, NN denotes the number of questions, and G=32G=32 responses are sampled per question, consistent with the number of rollouts used in TTCL. For each sampled answer aija_{ij}, q(aij∣xi)q(a_{ij}\mid x_i) denotes its frequency-based supervision target, while zij∈{0,1}z_{ij}\in\{0,1\} denotes its correctness. The metric lies in [0,1][0,1], with a lower value indicating better agreement between the supervision targets and observed correctness. We reported these results as follows:

Target-Correctness GapMATH500AMCAIME24AIME25DAPO-MathLightEval-MATHNQHotpotQATriviaQA
Qwen3-4B0.11980.17380.18620.19130.14710.10100.19310.33590.3257
Qwen3-8B0.15730.15750.18030.19110.21020.20750.21570.30880.2948
Llama-3.1-8B-Instruct0.19920.18480.07680.05570.15910.17940.25680.26600.2769

We observe that the Target-Correctness Gap remains within a relatively small range across the models and benchmarks. This provides a quantitative characterization of the discrepancy between frequency-based targets and observed correctness, supporting the use of answer frequency as a self-supervised signal in TTCL. These findings also provide empirical support for the assumption in our theoretical analysis that answer frequency serves as a proxy for the latent correctness distribution. Meanwhile, the assumption is also adopted in existing label-free self-supervised RL studies [1, 2, 3]. We consider this assumption reasonable to some extent because the broad knowledge acquired through large-scale pretraining can guide LLMs toward correct answers.

Moreover, we would like to clarify how TTCL adjusts confidence in incorrect answers. Although the correctness reward reinforces the majority answer, the calibration reward uses its cached frequency as a soft target, rather than assigning it a confidence target of 1. Consequently, when an incorrect answer’s frequency is lower than the model’s confidence, the calibration objective encourages a reduction in confidence. This is particularly relevant to the substantial over-confidence observed in the initial models. where their mean confidence considerably exceeds their accuracy:

ModelMATH500AMCAIME24AIME25
ACCMean ConfACCMean ConfACCMean ConfACCMean Conf
Qwen3-4B80.600.98459.380.98420.420.95116.250.979
Qwen3-8B84.400.98762.810.98321.670.97818.750.981
Llama-3.1-8B-Instruct39.600.85821.880.8524.580.8560.000.853
ModelSimpleQANQHotpotQATriviaQA
ACCMean ConfACCMean ConfACCMean ConfACCMean Conf
Qwen3-4B1.400.9278.630.9619.560.92715.630.954
Qwen3-8B2.200.9289.630.98312.750.95024.130.989
Llama-3.1-8B-Instruct3.200.84614.690.95412.310.85136.880.857

Thus, reinforcing an incorrect majority answer does not necessarily increase its confidence; when its empirical frequency is below the model’s predicted confidence, frequency-based supervision instead encourages a confidence reduction. Meanwhile, the target cache further prevents increases in the current policy’s answer consistency from immediately raising the confidence targets, mitigating the feedback loop that could reinforce an incorrect answer with increasingly high confidence. The training dynamics in Figures 5, 11, and 12 of the manuscript show that TTCL progressively narrows the gap between accuracy and mean confidence, providing empirical evidence that it improves the models’ calibration behavior and mitigates their overconfidence. We will update these analyses into the revised manuscript.

[1] TTRL: Test-time Reinforcement Learning. NeurIPS, 2025.
[2] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models. ICLR, 2026.
[3] Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to reviewer i6Lq (Part 2/4)

W2: Limited methodological novelty. Majority-vote pseudo-labeling, consistency-based confidence estimation, and confidence distillation have been explored in prior work. The contribution appears to lie mainly in combining these components for test-time adaptation.

A: Thanks for your valuable comments. We would like to clarify that our main contribution is to enable calibration learning directly on unlabeled target tasks at test time. Existing calibration learning methods typically rely on labeled source data, and the resulting calibration may transfer poorly to different target tasks, as illustrated in Fig. 2. TTCL addresses this problem by constructing self-supervision to jointly improve reasoning capability and confidence calibration without target-task annotations. Through TTCL adaptation, the model learns to directly express how likely its answer is to be correct, incorporating confidence estimation into its parameters and producing an answer together with verbalized confidence in a single response. We conduct extensive experiments across three models and more than ten benchmarks to evaluate its effectiveness.

Although majority-vote pseudo-labeling has been explored in existing label-free RL methods, these methods primarily focus on improving reasoning performance without consideration of optimizing confidence calibration. We evaluate models trained using several representative methods, including Self-Certainty Maximization [1], Entropy Minimization [2], Majority-Voting [3], and Co-rewarding [4]. The results are reported as follows:

ModelMethodMATH500AMCAIME24AIME25AIME26
ACC↑ECE↓ConfACC↑ECE↓ConfACC↑ECE↓ConfACC↑ECE↓ConfACC↑ECE↓Conf
Qwen3-4B-BaseBase Model58.80.3470.93538.440.5410.92610.000.8260.9264.170.8760.9186.250.8610.924
Self-Certainty61.60.3420.95841.250.5400.9537.500.8770.9525.000.8940.9445.000.9000.950
Entropy77.20.2060.97851.880.4620.98114.580.8330.9788.750.8900.9789.580.8850.980
Majority-voting72.60.2300.95643.130.5090.9409.580.8310.9274.580.8760.9215.830.8640.923
Co-rewarding75.60.2140.97052.810.4310.95914.580.8010.9479.170.8520.9446.670.8810.947
Qwen3-8B-BaseBase Model60.60.3050.91142.500.4870.91213.750.7730.91113.330.7780.9126.250.8460.908
Self-Certainty83.20.1320.96459.690.3630.96020.000.7500.95016.250.7870.94913.330.8200.954
Entropy79.00.1770.96757.500.3880.96314.170.8180.96014.170.8130.9559.170.8660.957
Majority-voting78.20.1800.96257.190.3800.95212.080.8220.9437.500.8620.9376.250.8750.937
Co-rewarding81.00.1570.96760.630.3500.95615.000.7970.94711.250.8340.94611.670.8290.945
Llama-3.2-3B-InstructBase Model35.20.5140.86624.060.6150.8556.670.7660.8320.000.8280.8280.830.8250.834
Self-Certainty47.00.4280.89825.000.6370.8876.670.8070.8740.420.8650.8690.420.8630.867
Entropy31.00.5890.89915.630.7300.8875.000.8150.8650.420.8660.8700.830.8500.858
Majority-voting42.00.4700.89025.630.6190.8757.500.7810.8560.420.8460.8500.830.8380.847
Co-rewarding46.40.4260.89028.440.5610.8467.080.7430.8140.000.8130.8131.250.8010.814

It can be observed that models trained by these methods exhibit substantial over-confidence, suggesting that improving reasoning through label-free RL alone does not ensure well-calibrated confidence. Our contribution is to address this gap by enabling models to jointly learn reasoning and calibrated confidence expression from unlabeled target-task data.

[This response to W2 continues in the next part.]

AuthorsOpenReview ↗

Response to reviewer i6Lq (Part 3/4)

[continued response to W2]

Additionally, TTCL jointly optimizes reasoning capability and confidence calibration in an unlabeled setting, posing additional challenges due to the lack of ground-truth supervision and potential interference between the two objectives during training. For example, when answer frequencies are used as confidence targets, reinforcing a majority answer can increase its frequency and consequently raise its confidence target, creating a feedback loop that can lead to reward hacking. Moreover, applying calibration rewards to the entire response can penalize a correct reasoning trajectory when its associated confidence deviates from the calibration target. The design of TTCL addresses these challenges through self-supervised reward construction, a target cache, and token-level advantage decoupling, while maintaining a relatively simple and intuitive algorithmic framework. We provide theoretical analysis that connects the TTCL objective to the ideal calibration objective through an error-dependent bound.

To intuitively clarify the contributions of TTCL, we summarize its distinctions from related methods below:

CategoryMethodLabel-freeCorrectness training signalCalibration objectiveConfidence MechanismModel training
Existing Calibration LearningRLCR [5]×Correctness reward√Learned verbalized confidence√
Reward Doubt [6]×None√Learned verbalized confidence√
Existing Label-free Confidence EstimationBaseCal [7]√None√Base-model probabilities×
Verbalization [8]√None√Prompted confidence×
Sample Consistency [9]√None√Answer consistency×
Existing Label-free RLSelf-Certainty Maximization [1]√Self-certainty reward×No explicit calibration objective√
Entropy Minimization [2]√Entropy-based reward×No explicit calibration objective√
Majority-Voting [3]√Majority-based reward×No explicit calibration objective√
Co-rewarding [4]√Majority-based reward×No explicit calibration objective√
Label-free Calibration Learning (Ours)TTCL√Majority-based reward√Learned verbalized confidence√

Overall, existing calibration learning methods typically rely on ground-truth annotations, while existing label-free RL methods primarily improve reasoning without explicitly addressing confidence calibration. TTCL bridges this gap by jointly optimizing reasoning capability and confidence calibration on unlabeled target-task data. Furthermore, existing label-free confidence estimation methods often obtain confidence scores through post-hoc computations. TTCL instead aims to internalize confidence estimation through unlabeled adaptation, enabling the model itself to express how likely its answer is to be correct alongside its response.

[1] Learning to Reason without External Rewards. ICLR, 2026.
[2] Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization. NeurIPS, 2025.
[3] TTRL: Test-time Reinforcement Learning. NeurIPS, 2025.
[4] Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models. ICLR, 2026.
[5] Beyond Binary Rewards: Training LMs to Reason about Their Uncertainty. ICLR, 2026.
[6] Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. ICLR, 2026.
[7] BaseCal: Unsupervised Confidence Calibration via Base Model Signals. ACL, 2026.
[8] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP, 2023.
[9] Calibrating Large Language Models with Sample Consistency. AAAI, 2025.

Reviewer_9aJmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To address the overconfidence of large language models in unlabeled test scenarios and cross-domain transfer, this paper proposes a Test-Time Calibration Learning (TTCL) framework, aiming to simultaneously improve the model’s reasoning accuracy and confidence calibration using unlabeled target data. The method samples multiple responses to the same question, extracting the majority-vote result and the empirical frequency of each answer as correctness pseudo-labels and calibration soft targets, respectively. It performs reinforcement learning updates by decoupling gradients for reasoning tokens and confidence tokens and introducing an exponential moving average (EMA) target cache. Effectiveness is validated on mathematical reasoning and factual question-answering tasks.

Critical strengths
  1. Clear Entry Point and Problem Motivation: The paper clearly reveals the degradation of existing supervised calibration methods under cross-domain transfer, as well as the tendency of conventional unlabeled reinforcement learning to make models blindly overconfident.

  2. Reasonable Design of Soft Targets and Anti-Collapse Mechanisms: Using continuous sampling frequencies as soft targets is significantly better than binary pseudo-labels. Meanwhile, token-level gradient decoupling and the lagged target cache effectively alleviate multi-objective optimization conflicts and self-training collapse.

Critical weaknesses
  1. The model repeatedly samples hundreds to thousands of times directly on the test sample set and performs multiple rounds of reinforcement learning, after which it is evaluated on the same set of samples. This neither supports online streaming inference nor distinguishes whether the model has learned a general calibration ability or simply overfit to specific test questions. It is recommended to strictly split the target-domain data into an unlabeled adaptation set and an independent test set, add generalization experiments on unseen samples, and report actual computational time and cost.

  2. Since the authors have already paid the high cost of multiple sampling on test samples, directly taking the majority vote of the initial model’s multiple samples as the answer and the empirical frequency as the confidence would achieve zero-training calibration. However, the paper does not compare against this most direct sampling-consistency baseline (e.g., related ideas such as Lyu et al., 2025). It is recommended to add this baseline to demonstrate the necessity of distilling frequencies into verbalized confidence through reinforcement learning.

  3. The theoretical derivation directly assumes that sampling frequency can accurately reflect the true correctness rate, ignoring collective hallucination and bias introduced by policy updates. In addition, the experiments lack a complete quantitative ablation of the target cache and token-level gradient decoupling. It is recommended to weaken the claims of theoretical guarantees and add overall metric changes after removing these two core modules.

Questions for authors

If the target-domain dataset is split in half into an unlabeled training subset and an independent test subset (running TTCL only on the training subset and performing single-pass inference evaluation on the unseen test subset), do the improvements in accuracy and calibration still hold?

10 comments
AuthorsOpenReview ↗

Response to reviewer 9aJm (Part 1/6)

W1: The model repeatedly samples hundreds to thousands of times directly on the test sample set and performs multiple rounds of reinforcement learning, after which it is evaluated on the same set of samples. This neither supports online streaming inference nor distinguishes whether the model has learned a general calibration ability or simply overfit to specific test questions. It is recommended to strictly split the target-domain data into an unlabeled adaptation set and an independent test set, add generalization experiments on unseen samples, and report actual computational time and cost.
Q1: If the target-domain dataset is split in half into an unlabeled training subset and an independent test subset (running TTCL only on the training subset and performing single-pass inference evaluation on the unseen test subset), do the improvements in accuracy and calibration still hold?

A: Thanks for your valuable comments. We would like to clarify that TTCL aims to enable models to learn to express their confidence directly on unlabeled target tasks, without ground-truth annotations. Existing calibration learning methods typically rely on labeled source data, while their calibration performance may transfer poorly to different target tasks, as illustrated in Fig. 2. This motivates TTCL, which adapts the model’s calibration behavior to the target task through self-supervised training. Through TTCL adaptation, the model learns to express how likely its answer is to be correct, incorporating this confidence estimation capability into its parameters. Related test-time learning settings have also been studied in prior work on LLMs. For example, TTRL [1] performs reinforcement learning using unlabeled test data, while TLM [2] updates LLMs using self-supervised objectives on unlabeled test inputs.

We agree with the reviewer that held-out experiments help assess whether this learned capability generalizes to unseen target-task examples. Following your suggestion, we supplement our experiments with a held-out setting that splits the target-task data into an unlabeled adaptation set and a disjoint test set. For mathematical reasoning, we randomly sample 20 examples from each of MATH500, AMC, AIME24, and AIME25 to form an unlabeled adaptation set of 80 examples, and use the remaining examples from each benchmark as the test set. For factual QA, we randomly sample 100 examples from each of SimpleQA, NQ, HotpotQA, and TriviaQA to form an unlabeled adaptation set of 400 examples, and use the remaining examples as the test set. After adaptation, we evaluate the model on the held-out test set, without further parameter updates on the test examples. The results are reported as follows:

[This response to W1 and Q1 continues in the next part.]

AuthorsOpenReview ↗

Response to reviewer 9aJm (Part 2/6)

[continued response to W1 and Q1]

The results are reported as follows:

ModelMethodMATH500AMCAIME24AIME25
ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑
Qwen3-4BBase Model80.210.1810.65967.500.3180.69218.750.7880.73318.750.7220.697
TTCL83.540.0990.88368.750.0750.81822.500.1850.85023.750.2680.939
#Improv+4.16%+45.48%+34.09%+1.85%+76.51%+18.25%+20.00%+76.52%+15.91%+26.67%+62.83%+34.66%
Qwen3-8BBase Model80.210.1520.59866.250.3100.57321.250.7740.51325.000.7410.619
TTCL83.130.0600.84570.630.1130.89325.000.2800.92327.500.2690.814
#Improv+3.64%+60.70%+41.24%+6.60%+63.49%+55.92%+17.65%+63.78%+79.85%+10.00%+63.66%+31.60%
Llama-3.1-8B-InstructBase Model39.580.4330.53120.630.6140.4462.500.8090.2710.000.865-
TTCL41.880.1770.71233.130.1540.6167.500.3270.7140.000.391-
#Improv+5.79%+59.12%+34.09%+60.59%+74.92%+38.12%+200%+59.58%+163.5%-+54.80%-
RLCR-calibrated Qwen3-4B on FactQA-10KBase Model80.210.1810.65967.500.3180.69218.750.7880.73318.750.7220.697
RLCR Model81.670.1170.76171.250.1430.65321.250.2670.81023.750.3310.839
TTCL82.290.0860.90575.000.1140.90622.500.1740.91426.250.1680.931
#Improv+0.76%+26.50%+18.92%+5.26%+20.28%+38.74%+5.88%+34.83%+12.84%+10.53%+49.24%+10.97%
ModelMethodSimpleQANQHotpotQATriviaQA
ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑ACC↑ECE↓AUROC↑
Qwen3-4BBase Model1.670.9070.4748.000.8630.7567.880.8240.66518.750.7620.674
TTCL2.000.2500.72910.500.2750.8368.500.2410.71920.380.2480.837
#Improv+19.76%+72.44%+53.80%+31.25%+68.13%+10.58%+7.87%+70.75%+8.12%+8.69%+67.45%+24.18%
RLCR-calibrated Qwen3-4B on DAPO-Math-14KBase Model1.670.9070.4748.000.8630.7567.880.8240.66518.750.7620.674
RLCR Model2.000.6100.65710.880.6590.7568.500.5610.80222.380.5280.729
TTCL2.670.3280.72513.630.3380.8249.130.3290.86923.380.2980.856
#Improv+33.50%+46.23%+10.35%+25.28%+48.71%+8.99%+7.41%+41.35%+8.35%+4.47%+43.56%+17.42%
RLCR-calibrated Qwen3-8B on DAPO-Math-14KBase Model2.330.8920.50910.250.8680.6268.380.8500.65826.500.7040.558
RLCR Model2.440.7590.61713.250.7310.6609.750.6190.66930.750.5540.708
TTCL2.780.5060.68015.250.4590.79010.620.4760.69232.380.3800.756
#Improv+13.93%+33.33%+10.21%+15.09%+37.21%+19.70%+8.92%+23.10%+3.44%+5.30%+31.41%+6.78%

From these results, we observe that TTCL generally improves accuracy while consistently reducing ECE and improving AUROC on the held-out test sets. These gains are observed for both the initial models and the RLCR-calibrated models, showing that TTCL remains effective when adaptation and evaluation examples are disjoint. Together, these results suggest that the confidence estimation capability learned through TTCL generalizes to unseen target-task examples, supporting the effectiveness of our TTCL in learning confidence expression through unlabeled adaptation.

[This response to W1 and Q1 continues in the next part.]

AuthorsOpenReview ↗

Response to reviewer 9aJm (Part 3/6)

[continued response to W1 and Q1]

Additionally, we agree with the reviewer that reporting computational costs would facilitate further analysis of TTCL. Following your suggestion, we report the number of training steps and training time for each model under the held-out setting described above. All experiments were conducted on 4xH100 GPUs.

ModelDatasetNumber of examplesTraining stepsTotal adaptation timeTime per step (s/step)
Qwen3-4BMath adaptation set80501 h 36 min 1 s115.22
Qwen3-8BMath adaptation set80502 h 29 min 6 s178.92
Llama-3.1-8B-InstructMath adaptation set80501 h 55 min 30 s138.60
RLCR-calibrated Qwen3-4B on DAPO-Math-14KFactQA adaptation set400602 h 54 min 58 s174.97
RLCR-calibrated Qwen3-8B on DAPO-Math-14KFactQA adaptation set400604 h 12 min 48 s252.80
RLCR-calibrated Qwen3-4B on FactQA-10KMath adaptation set80501 h 41 min 27 s121.74

These measurements, together with the held-out results, show that TTCL can improve calibration on unseen questions through 50-60 training steps on a small unlabeled adaptation set. Once adaptation is complete, the model can answer new questions and directly express its confidence in a single response, without requiring repeated sampling or additional post-hoc confidence estimation.

[1] TTRL: Test-Time Reinforcement Learning. NeurIPS, 2025.
[2] Test-Time Learning for Large Language Models. ICML, 2025.

Comments on the paper

2 comments

Dear Reviewers

Thank you for your time and effort throughout the review process. I would appreciate it if you could read the author's rebuttal and provide a response. Your participation in the discussion will help ensure a smooth decision process.

Thank you for your support. Best regards, Area Chair

AuthorsOpenReview ↗

General Response

Dear AC and All Reviewers,

We would like to express our deep gratitude to all reviewers for their contrustive suggestions and valuable comments, which are immensely helpful for us. From their review comments, we are pleased to receive numerous positive remarks, including:

  • Clear entry point and problem motivation: Reviewer 9aJm.
  • Practical importance and problem of label-free calibration under domain shift: Reviewers i6Lq, and XxFa.
  • Reasonable methodological design of frequency-based soft targets, token-level decoupling, and the lagged target cache: Reviewer 9aJm.
  • Sensible separation of answer and confidence optimization: Reviewers 9aJm and XxFa.
  • Extensive experiments demonstrating effectiveness: Reviewer 9aJm and i6Lq.

More importantly, following the reviewers’ suggestions and follow-up questions, we have incorporated additional experiments, analyses, and clarifications into the revised manuscript, improving the quality of the manuscript. We summarize the main updates below:

  • Held-out setting on unseen target-task examples: added in Section 5.2, Tables 6–7, and Appendix E.3.
  • Comparison with the Sample Consistency baseline: added in Section 5.3 and Table 10.
  • Joint analysis of frequency, confidence, and correctness: added in Section 4 and Table 1.
  • Clarification of the theoretical guarantee: revised in Section 4, Theorem 4.1, and Appendix D.3.
  • Ablations of the target cache and token-level gradient decoupling: added in Section 5.3 and Figure 4.
  • Confidence on correct and incorrect majority answers: added in Appendix F.9 and Table 23.
  • Analysis of sampling efficiency and diversity through pass@k and maj@k: added in Appendix F.7 and Figure 14.
  • Adaptation costs and inference-time comparisons with Sample Consistency: added in Appendix F.5 and Tables 21-22.

All revisions are highlighted in purple, including the captions of updated tables and figures. We sincerely thank all reviewers and the Area Chair for their valuable time and thoughtful feedback. Your constructive comments have helped us strengthen the manuscript and identify important directions for future research.

Our work is motivated by the practical need for label-free calibration on target tasks and supported by extensive experiments. Throughout the rebuttal period, we have actively engaged in the discussion, clarified unclear aspects, and conducted substantial additional experiments to address the reviewers’ concerns as thoroughly as possible. We have made every effort to strengthen the manuscript, clarify the supporting evidence, and address the reviewers’ concerns.

Considering the situation of this year’s rebuttal process, we would be sincerely grateful if you could take our contribution and the further evidence provided during the rebuttal period into consideration when making your final assessment.

Best regards,

Authors of 24042

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.