- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RLICA, which extends RLCR (GRPO with correctness, format and Brier rewards on a verbalized confidence) with a reward that pulls the stated confidence toward an internal estimate of correctness. The estimate is a hidden-state ridge probe for semantic entropy, mapped to a correctness probability by an isotonic fit, both refit periodically on policy rollouts. Training Qwen3.5-4B Base with LoRA on a modified HotpotQA, one run per method, the authors report better out-of-distribution accuracy and calibration than RLCR, and a stated confidence that drops when the model's own reasoning expresses doubt far more often (88% against 31% out of distribution). Abstention, sycophancy, probe-drift and steering analyses follow.
Critical strengths
-
The paper tests something aggregate calibration cannot see. Asking whether the stated confidence drops when the model's own analysis expresses doubt targets a failure that ECE hides, and the effect here is large rather than marginal. This is the clearest evidence that the alignment term changes how the model reports confidence.
-
The comparison with RLCR is clean and the reporting is transparent. The two methods share one configuration and differ only in the alignment term, differences come with paired bootstrap intervals, per-dataset results are complete, and a frozen-map ablation shows that the online refit matters. This makes the open questions below easy to locate.
Critical weaknesses
-
The experiments do not isolate the mechanism, so the claim that the gains come from internal uncertainty is not yet supported. Because the isotonic map is fit on correctness, the internal estimate is in effect a calibrated correctness predictor, and any such predictor could serve as the target; no control keeps a calibrated target while removing the hidden-state information, and only one alignment weight is tried. The claim that the model states a confidence equal to its accuracy when its samples disagree is likewise untested: only aggregate calibration is reported, and the steering experiment uses the frozen base model rather than the trained policy. The proper-scoring-rule framing also overreaches: the combined reward peaks at a weighted average of the correctness probability and the internal estimate, pulling confidence away from the truth where the probe errs, which fits the significant out-of-distribution AUROC drop the abstract omits. The paper's central attribution to hidden states therefore remains open.
-
The headline gains are small, rest on a single training seed, and are concentrated in individual datasets. Each method is trained once, and the bootstrap intervals resample evaluation prompts, so they capture test-set noise, not run-to-run variation; in distribution, only AUROC differs significantly. Out of distribution, most of the accuracy gain comes from GPQA, where RLCR falls below even RLVR, while SimpleQA alone accounts for more than the entire ECE improvement: without it, RLICA's mean ECE is worse than RLCR's, as on most out-of-distribution sets. GPQA also carries nearly all of the AUROC loss. Inconsistencies in the text, such as 312 versus 208 training steps and comparisons with RLCR labeled as comparisons with RLVR, obscure which runs the tables describe. Robustness across datasets and training runs is therefore not established.
Questions for authors
- Can you report RLICA with the internal estimate replaced by the empirical accuracy of the rollouts already sampled for the same prompt, all else fixed, plus a sweep over the alignment weight tracking out-of-distribution AUROC?
- What direct evidence supports the claim that stated confidence equals accuracy when samples disagree, for instance a reliability curve on the highest semantic-entropy bins?
- Which step count, 312 or 208, produced the evaluated checkpoints, and do the out-of-distribution gains, including RLCR's low GPQA accuracy, recur over at least two more seeds?
- Would you narrow the novelty claim in Section 3 to RL with a refit semantic-entropy probe and discuss Ji et al., "Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations" (EMNLP 2025), which finds a linear hidden-state feature governing verbal uncertainty?