- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CoCal, a framework for confidence calibration during reinforcement learning from verifiable rewards (RLVR). Instead of jointly optimizing task performance and confidence within the same policy, CoCal trains a lightweight companion model using hidden representations and correctness feedback collected from RLVR rollouts, while leaving policy optimization unchanged. Experiments on Qwen3-8B and Qwen3-14B across mathematical reasoning benchmarks show that CoCal improves confidence estimation compared with existing RL-based concurrent methods and matched post-hoc calibration, while preserving task performance. The paper further demonstrates promising cross-domain and cross-policy transfer, as well as low additional computational overhead.
Critical strengths
The main strength of this paper is its simple yet practical approach to confidence calibration. By reusing existing RLVR rollouts and separating confidence learning from policy optimization, CoCal avoids additional response sampling and preserves the original task performance. The empirical results are encouraging, with improvements over RLCR and DCPO in both calibration quality and task accuracy. Another important strength is the comparison with matched post-hoc calibration and the evaluation of cross-domain transfer. In particular, the improvements on factual QA benchmarks suggest that learning confidence from historical RLVR experience can provide useful generalization beyond the training domain. These findings, together with the low additional computational cost, make the approach practically valuable.
Critical weaknesses
My main concern is the limited methodological novelty and insufficient evidence explaining the reported improvements. CoCal employs a standard supervised MLP trained on hidden representations and correctness labels, closely resembling existing representation-based confidence estimation methods. Its main contribution therefore appears to be reusing historical RLVR trajectories rather than introducing a fundamentally new calibration mechanism. Moreover, since CoCal is trained through offline chronological replay, it remains unclear whether this procedure offers advantages over post-hoc training on the same trajectories. Although the comparison with PostCal matches raw sampling budgets, differences in training data distributions, particularly the higher proportion of incorrect responses in CoCal (Figure 10), may partly explain the improvements. Finally, the lower task accuracy of RLCR and DCPO does not conclusively demonstrate optimization interference, as these methods also differ in reward design and generation formats. These issues make it difficult to isolate CoCal's distinct methodological contribution despite its promising empirical results.
Questions for authors
Could the authors clarify the specific benefit of chronological learning? In particular, how would CoCal perform if the same cached RLVR rollouts were randomly shuffled before training the companion, rather than replayed chronologically? Additionally, could the authors provide a more controlled comparison with PostCal by matching the effective training sample sizes and the proportions of correct and incorrect responses, to determine whether the improvements mainly arise from historical policy diversity or differences in data composition? Finally, could the authors provide further evidence supporting the optimization interference hypothesis? Since RLCR and DCPO also differ in their reward formulations and confidence-generation formats, it would be helpful to clarify whether shared-parameter optimization is indeed the primary cause of their lower task accuracy.