- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TTCL adapts LLM reasoning and confidence on unlabeled target tasks. It derives rewards from sampled-answer agreement and frequencies, with evaluation on mathematical reasoning and factual QA.
Critical strengths
Label-free calibration under domain shift is a useful problem. Separating answer and confidence rewards is a sensible design choice.
Critical weaknesses
-
The calibration results contain a substantial inconsistency. For Qwen3-4B with TTCL on NQ, Tables 2 and 11 report accuracy 0.1063, mean confidence 0.420, and ECE 0.155. Under Eq. (3), ECE must be at least |0.420 − 0.1063| = 0.3137 if these use the same predictions.
-
The surrogate guarantee depends on an unquantified gap between answer frequency and correctness. Frequent but incorrect answers remain a fundamental concern.
-
The benefit of test-time RL over directly using answer frequencies as confidence scores is unclear, especially given the cost of 10–40 training epochs.
Questions for authors
-
Do Tables 2 and 11 use the same predictions and checkpoints? Please reconcile the reported metrics.
-
Can you quantify the frequency–correctness gap and evaluate frequent incorrect answers with the target cache enabled?
-
Could you compare TTCL with direct frequency-based confidence under matched compute budgets and report the adaptation time?
12 comments
Response to reviewer XxFa (Part 2/4)
W3: The benefit of test-time RL over directly using answer frequencies as confidence scores is unclear, especially given the cost of 10-40 training epochs.
Q3: Could you compare TTCL with direct frequency-based confidence under matched compute budgets and report the adaptation time?
A: Thanks for your valuable comments. We would like to clarify that our motivation is to enable the model to learn to express its own confidence without ground-truth annotations. Through TTCL, the model learns to directly express how likely its answer is to be correct, incorporating this confidence estimation capability into its parameters. Although TTCL requires multiple rollouts for a question, these are used during adaptation to train the model’s capability of confidence expression. After adaptation, the model can produce an answer and its verbalized confidence in a single inference pass.
Following your suggestions, we futher compare TTCL with the frequency-based confidence estimation, referred as SampleConsistency, which samples multiple rollouts and uses the majority answer as the prediction and frequency as the confidence. We use the number of multiple rollouts , matching the rollout group size used during TTCL adaptation. We report Pass@1, Avg@32, Maj@32 and ECE, where Pass@1 and Avg@32 assess the sampling efficiency, Maj@32 examines whether learning from frequency-based supervision improves majority-vote accuracy over directly using the initial model’s sampling results, while ECE measures calibration performance. The results are reported below:
| Model | Method | MATH500 | AMC | AIME24 | AIME25 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pass@1 ↑ | Avg@32 ↑ | Maj@32 ↑ | ECE ↓ | Pass@1 ↑ | Avg@32 ↑ | Maj@32 ↑ | ECE ↓ | Pass@1 ↑ | Avg@32 ↑ | Maj@32 ↑ | ECE ↓ | Pass@1 ↑ | Avg@32 ↑ | Maj@32 ↑ | ECE ↓ | ||
| Qwen3-4B | SampleConsistency | 79.00 | 81.34 | 87.60 | 0.041 | 62.50 | 65.08 | 80.00 | 0.083 | 20.00 | 19.58 | 26.67 | 0.119 | 13.33 | 17.50 | 23.33 | 0.147 |
| TTCL | 87.40 | 86.84 | 90.00 | 0.022 | 80.00 | 78.98 | 82.50 | 0.089 | 30.00 | 28.23 | 30.00 | 0.068 | 23.33 | 21.35 | 23.33 | 0.245 | |
| Qwen3-8B | SampleConsistency | 77.60 | 77.78 | 87.80 | 0.084 | 67.50 | 65.31 | 80.00 | 0.075 | 20.00 | 24.58 | 36.67 | 0.057 | 20.00 | 18.44 | 23.33 | 0.111 |
| TTCL | 88.80 | 88.66 | 90.20 | 0.082 | 77.50 | 79.38 | 85.00 | 0.103 | 26.67 | 29.58 | 40.00 | 0.073 | 26.67 | 26.15 | 26.67 | 0.109 | |
| Llama-3.1-8B-Instruct | SampleConsistency | 42.20 | 40.66 | 56.60 | 0.062 | 15.00 | 19.53 | 27.50 | 0.078 | 0.00 | 3.54 | 6.67 | 0.157 | 0.00 | 0.00 | 0.00 | 0.096 |
| TTCL | 56.40 | 55.38 | 62.60 | 0.101 | 42.50 | 43.75 | 50.00 | 0.109 | 10.00 | 9.48 | 10.00 | 0.216 | 0.00 | 0.00 | 0.00 | 0.114 |
From these results, we observe that TTCL generally improves Pass@1 and Avg@32, indicating better sampling efficiency. Its improved Maj@32 further shows that learning from frequency-based supervision can enhance the model’s answer distribution beyond the initial model’s voting performance. Even in some cases, TTCL’s single-response accuracy even exceeds the initial model’s majority-vote accuracy. For calibration, TTCL achieves lower or comparable ECE to the initial model’s frequency-based confidence estimates. These results suggest that TTCL adaptation enables the model to internalize confidence estimation from multiple rollouts and express it directly through verbalized confidence, allowing the model to assess the correctness of its answer in a single response without repeated sampling at inference time. This reduces the number of sampled responses per question from K to one, yielding an approximately K-fold reduction in inference-time sampling cost.
[This response continues in the next part.]
Response to reviewer XxFa (Part 3/4)
[continued response to W3 and Q3]
Additionally, to examine whether this learned capability generalizes to unseen questions, we supplement the experiments with a held-out setting, where the target-task data are divided into an unlabeled adaptation set and a disjoint test set. TTCL is trained only on the adaptation set and evaluated on the test set. Specifically, for mathematical reasoning, we randomly sample 20 examples from each of MATH500, AMC, AIME24, and AIME25 to form an unlabeled adaptation set of 80 examples for TTCL training, and use the remaining examples from each benchmark for evaluation; for factual QA, we randomly sample 100 examples from each of SimpleQA, NQ, HotpotQA, and TriviaQA as the unlabeled adaptation set of 400 examples for TTCL training, and use the remaining examples for evaluation. The results are reported below:
| Model | Method | MATH500 | AMC | AIME24 | AIME25 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ||
| Qwen3-4B | Base Model | 80.21 | 0.181 | 0.659 | 67.50 | 0.318 | 0.692 | 18.75 | 0.788 | 0.733 | 18.75 | 0.722 | 0.697 |
| TTCL | 83.54 | 0.099 | 0.883 | 68.75 | 0.075 | 0.818 | 22.50 | 0.185 | 0.850 | 23.75 | 0.268 | 0.939 | |
| #Improv | +4.16% | +45.48% | +34.09% | +1.85% | +76.51% | +18.25% | +20.00% | +76.52% | +15.91% | +26.67% | +62.83% | +34.66% | |
| Qwen3-8B | Base Model | 80.21 | 0.152 | 0.598 | 66.25 | 0.310 | 0.573 | 21.25 | 0.774 | 0.513 | 25.00 | 0.741 | 0.619 |
| TTCL | 83.13 | 0.060 | 0.845 | 70.63 | 0.113 | 0.893 | 25.00 | 0.280 | 0.923 | 27.50 | 0.269 | 0.814 | |
| #Improv | +3.64% | +60.70% | +41.24% | +6.60% | +63.49% | +55.92% | +17.65% | +63.78% | +79.85% | +10.00% | +63.66% | +31.60% | |
| Llama-3.1-8B-Instruct | Base Model | 39.58 | 0.433 | 0.531 | 20.63 | 0.614 | 0.446 | 2.50 | 0.809 | 0.271 | 0.00 | 0.865 | - |
| TTCL | 41.88 | 0.177 | 0.712 | 33.13 | 0.154 | 0.616 | 7.50 | 0.327 | 0.714 | 0.00 | 0.391 | - | |
| #Improv | +5.79% | +59.12% | +34.09% | +60.59% | +74.92% | +38.12% | +200% | +59.58% | +163.5% | - | +54.80% | - | |
| RLCR-calibrated Qwen3-4B on FactQA-10K | Base Model | 80.21 | 0.181 | 0.659 | 67.50 | 0.318 | 0.692 | 18.75 | 0.788 | 0.733 | 18.75 | 0.722 | 0.697 |
| RLCR Model | 81.67 | 0.117 | 0.761 | 71.25 | 0.143 | 0.653 | 21.25 | 0.267 | 0.810 | 23.75 | 0.331 | 0.839 | |
| TTCL | 82.29 | 0.086 | 0.905 | 75.00 | 0.114 | 0.906 | 22.50 | 0.174 | 0.914 | 26.25 | 0.168 | 0.931 | |
| #Improv | +0.76% | +26.50% | +18.92% | +5.26% | +20.28% | +38.74% | +5.88% | +34.83% | +12.84% | +10.53% | +49.24% | +10.97% |
| Model | Method | SimpleQA | NQ | HotpotQA | TriviaQA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ACC↑ | ECE↓ | AUROC↑ | ||
| Qwen3-4B | Base Model | 1.67 | 0.907 | 0.474 | 8.00 | 0.863 | 0.756 | 7.88 | 0.824 | 0.665 | 18.75 | 0.762 | 0.674 |
| TTCL | 2.00 | 0.250 | 0.729 | 10.50 | 0.275 | 0.836 | 8.50 | 0.241 | 0.719 | 20.38 | 0.248 | 0.837 | |
| #Improv | +19.76% | +72.44% | +53.80% | +31.25% | +68.13% | +10.58% | +7.87% | +70.75% | +8.12% | +8.69% | +67.45% | +24.18% | |
| RLCR-calibrated Qwen3-4B on DAPO-Math-14K | Base Model | 1.67 | 0.907 | 0.474 | 8.00 | 0.863 | 0.756 | 7.88 | 0.824 | 0.665 | 18.75 | 0.762 | 0.674 |
| RLCR Model | 2.00 | 0.610 | 0.657 | 10.88 | 0.659 | 0.756 | 8.50 | 0.561 | 0.802 | 22.38 | 0.528 | 0.729 | |
| TTCL | 2.67 | 0.328 | 0.725 | 13.63 | 0.338 | 0.824 | 9.13 | 0.329 | 0.869 | 23.38 | 0.298 | 0.856 | |
| #Improv | +33.50% | +46.23% | +10.35% | +25.28% | +48.71% | +8.99% | +7.41% | +41.35% | +8.35% | +4.47% | +43.56% | +17.42% | |
| RLCR-calibrated Qwen3-8B on DAPO-Math-14K | Base Model | 2.33 | 0.892 | 0.509 | 10.25 | 0.868 | 0.626 | 8.38 | 0.850 | 0.658 | 26.50 | 0.704 | 0.558 |
| RLCR Model | 2.44 | 0.759 | 0.617 | 13.25 | 0.731 | 0.660 | 9.75 | 0.619 | 0.669 | 30.75 | 0.554 | 0.708 | |
| TTCL | 2.78 | 0.506 | 0.680 | 15.25 | 0.459 | 0.790 | 10.62 | 0.476 | 0.692 | 32.38 | 0.380 | 0.756 | |
| #Improv | +13.93% | +33.33% | +10.21% | +15.09% | +37.21% | +19.70% | +8.92% | +23.10% | +3.44% | +5.30% | +31.41% | +6.78% |
From these results, we observe that TTCL generally improves accuracy while consistently reducing ECE and improving AUROC on the held-out test sets. These gains are observed for both the initial models and the RLCR-calibrated models, indicating that TTCL remains effective when adaptation and evaluation examples are disjoint. In particular, TTCL further improves models calibrated on a different source domain, including FactQA-calibrated models evaluated on mathematical reasoning and math-calibrated models evaluated on factual QA. Together, these results suggest that the confidence estimation capability learned through TTCL generalizes to unseen target-task examples.
[This response continues in the next part.]
Response to reviewer XxFa (Part 1/4)
A: Thanks for your valuable comments. We agree that you are correct that the originally reported values cannot describe the same set of predictions. We would like to clarify that the inconsistency arose when we updated the tables using results from a later checkpoint. Specifically, we updated ACC, ECE, Brier score, and AUROC using the step-60 checkpoint, but inadvertently retained the mean confidence of 0.420 from the step-40 checkpoint. The corresponding mean confidence for the step-60 checkpoint is 0.221. We apologize for this reporting error. For transparency, we provide the evaluation results from both checkpoints below:
Thus, the original entries in Tables 2 and 11 inadvertently combined metrics from different checkpoints. The corrected entry consistently reports the step-60 results, replacing 0.420 with 0.221 in Table 11 while retaining the ACC, ECE, Brier score, and AUROC listed above.
A: Thanks for your contrustive comments. We agree that quantifying the discrepancy between frequency-based supervision and correctness is important for interpreting the surrogate bound. Theorem 4.1 connects the TTCL objective to the ideal calibration objective through the frequency-correctness estimation error. Following your suggestions, to provide a quantitative analysis, we introduce the Target-Correctness Gap, which measures the mean absolute difference between each sampled answer’s frequency and its observed correctness:
Target-Correctness Gap=NG1∑i=1N∑j=1G∣q(aij∣xi)−zij∣,
where N is the number of questions, G is the number of responses sampled per question (G=32 here), q(aij∣xi) is the cached frequency of answer aij, and zij∈{0,1} indicates whether that answer is correct. This metric ranges from 0 to 1, with smaller values indicating closer agreement between answer frequency and observed correctness. Ground-truth answers are used only for this post-hoc analysis and are not accessed during TTCL training. We examine this metric across multiple models and benchmarks, with the results reported below:
From these results, we observe that the values of Target–Correctness Gap remain within a relatively small range. This metric characterizes the discrepancy between frequency-based self-supervision targets and observed correctness, providing an empirical diagnostic related to the estimation error in Theorem 4.1. These results suggest that answer frequency can serve as a useful proxy for latent correctness. Together with our theoretical analysis, they indicate that TTCL provides a reasonable surrogate for calibration learning in the absence of ground-truth labels.
Complementing this analysis, the training dynamics in Figures 5, 11, and 12 in the manuscript show that TTCL progressively narrows the gap between mean confidence and empirical accuracy. This provides evidence that optimizing the TTCL objective improves the alignment between the model’s expressed confidence and its actual accuracy. Taken together, the quantified frequency-correctness gap, theoretical analysis, extensive experiments, and observed training dynamics support both the rationale for TTCL’s calibration objective and its effectiveness in improving confidence estimation of the model. We will update these results and analyses into the revised manuscript.