acceptodds
Under review as a conference paper at ICLR 2027

Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs

Abstract

Reliable self-assessment is essential for large language models (LLMs), yet they often remain highly confident when their answers are wrong. We study concurrent confidence calibration, where confidence is learned alongside capability improvement rather than calibrated only after training. Reinforcement learning from verifiable rewards (RLVR) provides a natural setting for this paradigm, as it continuously produces responses paired with verifiable correctness feedback. Existing concurrent methods, however, learn both capability and confidence through reinforcement learning within shared policy parameters, potentially coupling two fundamentally different learning problems. We instead propose shared experience but separate learning: capability and confidence learn from the same trajectories, but through separate optimization mechanisms and parameters. Based on this principle, we introduce CoCal (Companion Confidence Calibration), which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged. Experiments on Qwen3-8B and Qwen3-14B show that CoCal improves confidence estimation without sacrificing task performance, outperforming both RL-based concurrent methods and matched post-hoc calibration. The learned companion further generalizes across domains and policy shifts, while the benefits of CoCal persist at both scales.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Honesty, Confidence Calibration, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fQkfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCal, a framework for confidence calibration during reinforcement learning from verifiable rewards (RLVR). Instead of jointly optimizing task performance and confidence within the same policy, CoCal trains a lightweight companion model using hidden representations and correctness feedback collected from RLVR rollouts, while leaving policy optimization unchanged. Experiments on Qwen3-8B and Qwen3-14B across mathematical reasoning benchmarks show that CoCal improves confidence estimation compared with existing RL-based concurrent methods and matched post-hoc calibration, while preserving task performance. The paper further demonstrates promising cross-domain and cross-policy transfer, as well as low additional computational overhead.

Critical strengths

The main strength of this paper is its simple yet practical approach to confidence calibration. By reusing existing RLVR rollouts and separating confidence learning from policy optimization, CoCal avoids additional response sampling and preserves the original task performance. The empirical results are encouraging, with improvements over RLCR and DCPO in both calibration quality and task accuracy. Another important strength is the comparison with matched post-hoc calibration and the evaluation of cross-domain transfer. In particular, the improvements on factual QA benchmarks suggest that learning confidence from historical RLVR experience can provide useful generalization beyond the training domain. These findings, together with the low additional computational cost, make the approach practically valuable.

Critical weaknesses

My main concern is the limited methodological novelty and insufficient evidence explaining the reported improvements. CoCal employs a standard supervised MLP trained on hidden representations and correctness labels, closely resembling existing representation-based confidence estimation methods. Its main contribution therefore appears to be reusing historical RLVR trajectories rather than introducing a fundamentally new calibration mechanism. Moreover, since CoCal is trained through offline chronological replay, it remains unclear whether this procedure offers advantages over post-hoc training on the same trajectories. Although the comparison with PostCal matches raw sampling budgets, differences in training data distributions, particularly the higher proportion of incorrect responses in CoCal (Figure 10), may partly explain the improvements. Finally, the lower task accuracy of RLCR and DCPO does not conclusively demonstrate optimization interference, as these methods also differ in reward design and generation formats. These issues make it difficult to isolate CoCal's distinct methodological contribution despite its promising empirical results.

Questions for authors

Could the authors clarify the specific benefit of chronological learning? In particular, how would CoCal perform if the same cached RLVR rollouts were randomly shuffled before training the companion, rather than replayed chronologically? Additionally, could the authors provide a more controlled comparison with PostCal by matching the effective training sample sizes and the proportions of correct and incorrect responses, to determine whether the improvements mainly arise from historical policy diversity or differences in data composition? Finally, could the authors provide further evidence supporting the optimization interference hypothesis? Since RLCR and DCPO also differ in their reward formulations and confidence-generation formats, it would be helpful to clarify whether shared-parameter optimization is indeed the primary cause of their lower task accuracy.

Reviewer_uRF8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoCal, a confidence predictor trained using hidden representations and verifier feedback collected during RLVR. A separate MLP estimates confidence while the language model continues task training with GRPO. The predictor combines final-token representations with mean-pooled representations and is trained against either binary correctness targets or correctness targets weighted by rollout-group accuracy. In the implementation, the companion is trained through chronological replay of cached RL experience. The experiments continue RL training of Qwen3-8B and Qwen3-14B on DeepScaleR and evaluate confidence estimation on mathematical reasoning and factual QA, as well as transfer across policy checkpoints and training objectives.

Critical strengths
  1. Reliable confidence estimation during capability training is an important problem. Reusing existing verifier feedback provides a practical source of supervision, and the separate predictor preserves the task-policy update by construction.

  2. Comparisons with post-hoc and RL-based calibration, together with mathematics-to-factual-QA and policy-transfer evaluations, offer useful evidence within the studied setting.

Critical weaknesses
  1. The methodological novelty is limited, and the case for complete separation is not sufficiently established. SAPLMA already trains an independent MLP on frozen LLM activations [1], and Ni et al. employ a four-layer correctness estimator with last-token or mean-pooled representations [2]. More critically, CARE-PPO trains an actor with PPO while concurrently training a separate critic through squared-error regression, then explicitly uses that critic as a confidence estimator [7, Section 3.1, Eq. (3); Section 4.4]. This provides a direct precedent for learning an external confidence estimator alongside capability training, rather than fitting it only after RL. RLCR also includes an RLVR hidden-state probe [3], while DCPO already separates correctness and confidence objectives and uses group-level supervision [4]. CoCal discusses the latter literature but does not address CARE-PPO. Its remaining contribution needs to be distinguished from the combination of established hidden-state prediction and concurrent outcome regression: reusing actor representations in a small MLP, leaving GRPO unchanged, and selecting particular targets are concrete differences, but do not by themselves establish a new learning principle. Moreover, the comparisons with RLCR and DCPO vary the targets, losses, and output format as well as parameter sharing. Blocking confidence gradients preserves task training by construction; without a matched-head comparison, however, this does not establish that suitable shared auxiliary learning would be harmful.

  2. The experiments remain a restricted proof of concept and do not establish applicability to realistic industrial RL settings. Both backbones belong to the Qwen3 family, operate in non-thinking mode, and undergo 120 steps of continued RL on a single mathematics dataset, DeepScaleR. Evaluation covers mathematics and factual QA, demonstrating a particular cross-domain transfer but not generalization across heterogeneous task types or operational workflows. The study provides no mixed-task RL evaluation spanning, for example, mathematical reasoning, code execution, and multi-turn tool use; nor does it establish behavior across substantially larger model scales or longer RL runs. These omissions are particularly important for a companion trained alongside changing capabilities: task proportions, representation distributions, success criteria, and feedback quality can evolve together in industrial training. In such settings, a predictor that works on clean, final-answer correctness labels may instead track domain composition or verifier acceptance. The current experiments do not show whether a single companion remains calibrated within each domain as the training mixture changes, or whether domain-specific refitting is required. Broader practical relevance therefore remains unverified; the reported mathematical-RL and factual-QA results are insufficient to establish industrial generality.

  3. The supervision and evaluation require further qualification. CoCal (group) learns y_i multiplied by rollout-group accuracy. For eight conditionally IID responses with success probability p, the expected target is p/8 + 7p²/8 rather than p, so interpreting it as a response-correctness probability requires justification. Separately, the Verbal results in Table 1—a MATH500 accuracy of 0.852, a mean confidence of 0.997, and an ECE of 0.059—cannot all describe the same evaluation population. Appendix A.2 allows for this discrepancy by excluding invalid confidence outputs from calibration metrics while retaining them when computing accuracy. This is not a demonstrated arithmetic error, but the absence of coverage and valid-subset accuracy prevents an assessment of how this selection affects the comparisons.

[1] The Internal State of an LLM Knows When It’s Lying.

[2] Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception.

[3] Beyond Binary Rewards: Training LMs to Reason about Their Uncertainty.

[4] Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards.

[5] Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs.

[6] Language Models (Mostly) Know What They Know.

Questions for authors
  1. Apart from predictor cost and GRPO compatibility, how does CoCal advance beyond concurrent critic-based confidence learning [3] and hidden-state MLPs [1; 2]?

  2. With the head, confidence target, and output format held fixed, does blocking calibration gradients improve the accuracy–calibration frontier compared with allowing them into the policy?

  3. As task proportions change, can a single companion remain calibrated across mathematical, coding, and tool-use RL without domain-specific refitting?

[1] The Internal State of an LLM Knows When It’s Lying.

[2] Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception.

[3] From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation.

Reviewer_vL72OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCal, a confidence calibration framework that trains an independent MLP on hidden representations and correctness labels from RLVR rollouts, without modifying task-policy optimization. Experiments on mathematical reasoning and factual QA show that CoCal improves calibration while largely preserving task accuracy. CoCal also improves calibration in out-of-domain tasks. Comparisons suggest that multi-stage RLVR rollouts provide useful training data.

Critical strengths
  • CoCal separates confidence estimation from task-policy optimization by training an independent predictor on hidden representations and correctness labels from RLVR rollouts. This design avoids directly introducing calibration objectives into policy-gradient optimization, reducing the risk that improving confidence calibration will interfere with task accuracy. The approach is conceptually simple and potentially compatible with existing RLVR training pipelines.

  • The evaluation covers multiple mathematical reasoning benchmarks, two model scales, and additional factual question-answering tasks. The comparisons include both RL-based calibration methods and PostCal, which uses the same calibrator architecture and supervision but is trained on responses resampled from the final policy. The matched-data-budget experiments provide meaningful evidence that multi-stage RLVR rollouts can be more effective than final-policy resampling, with CoCal outperforming PostCal in most reported budget settings.

Critical weaknesses
  • My main concern is limited methodological novelty. CoCal essentially trains a separate confidence head on LLM hidden states, following established probing approaches. Its main distinction appears to be reusing rollouts across RLVR training stages rather than collecting data from the final policy. While practically useful, the current evidence does not clearly isolate whether the gains come from historical policy diversity, a higher proportion of incorrect responses, or chronological training.
  • In the comparition of CoCal and PostCal, CoCal uses rollouts from multiple training stages with a higher fraction of incorrect responses, so its gains may come from data composition and error diversity rather than concurrent calibration itself.
  • Experiments are restricted to Qwen3-8B and Qwen3-14B in non-thinking mode, with RLVR training focused on mathematics. It remains unclear whether the method transfers to other model families, thinking-mode reasoning, or non-mathematical RLVR tasks.
Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.