acceptodds
Withdrawn from ICLR 2027

Competence Before Generation, Confidence After: A Two-Stage Self-awareness Framework for Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While demonstrating remarkable capabilities, Large Language Models (LLMs) often generate fluent yet unreliable answers, making reliability estimation a critical prerequisite for safe deployment. Existing confidence estimators draw on surface-level outputs (e.g., token probabilities and multi-sample consistency) and internal-state probes. We study how internal reliability signals support prospective judgments before generation and retrospective judgments about a generated answer. This perspective echoes human meta-cognition, where monitoring at different stages supports self-awareness. Motivated by this insight, we propose a two-stage self-awareness framework that monitors model reliability from the internal states of a frozen language model. Before generation, the framework estimates the model's competence to answer correctly, defined through sampled-answer correctness. After generation, it assesses the correctness of the particular answer produced. Lightweight heads learn these stage-specific targets with sample-derived soft correctness and consistency supervision, and reuse a single autoregressive generation process at inference. Experiments on seven benchmarks demonstrate improvements in average ranking and calibration over the knowledge-boundary baseline under both in-domain and out-of-domain settings. Across nine in-domain settings, SA improves macro AUROC by 7.51 points; strict out-of-domain evaluation yields 0.7002 ± 0.0064 over twelve settings. Further analyses reveal complementary predictive information in the two stages: pre-generation competence remains informative about correctness after conditioning on post-generation confidence in pooled diagnostics. These findings highlight the value of stage-specific internal monitoring for LLM reliability estimation. Code will be made publicly available to facilitate future research.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Large Language Models, Confidence Estimation, Hidden-State Probing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ubjiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage self-awareness framework for LLM reliability estimation. It trains two lightweight probes on a frozen LLM's hidden states to predict pre-generation competence and post-generation answer correctness, respectively. Sample-derived correctness and consistency signals are used as auxiliary supervision. Experiments across multiple models and QA benchmarks demonstrate improvements in confidence estimation and calibration over the evaluated baselines.

Critical strengths
  1. Important problem and encouraging empirical results. Reliable confidence estimation is important for LLM deployment. The paper evaluates multiple model families across in-distribution and out-of-distribution settings, with generally promising improvements in confidence estimation and calibration.
  2. Useful empirical analyses. The paper includes ablations on auxiliary supervision, feature selection, and stage-specific confidence signals, providing some insights into the factors affecting reliability estimation.
Critical weaknesses
  1. Limited novelty and insufficient baseline comparisons. Prior work already uses pre- and post-generation hidden-state probes for reliability estimation. This paper distinguishes stage-specific prediction targets and adds sample-derived auxiliary supervision, but its methodological advance over existing probing approaches remains modest. Moreover, SA and the KB baseline use different probe architectures and optimization settings. Although the supervision ablations isolate the benefit of auxiliary targets within the SA configuration, they do not explain the substantial gap between KB and hard-only Post. Thus, the sources of the reported gains over KB remain insufficiently resolved.
  2. Unclear benefit of the two-stage design and limited evaluation. The main SA results rely only on the Post head, while combining Pre and Post does not consistently improve performance. This raises questions about the necessity of the two-stage framework. Moreover, the evaluation is limited to static QA tasks, without considering more realistic agentic scenarios involving tool use and multi-step decisions.

I would be open to raising my score if the authors can satisfactorily address these concerns and questions.

Questions for authors
  1. Given the existing supervision ablations, could the authors explain the substantial AUROC gap between KB and hard-only Post? A controlled comparison varying probe architecture and auxiliary supervision under a common optimization and model-selection protocol would help distinguish architectural and training effects from the benefit of the proposed supervision.
  2. What is the actual benefit of the Pre head if the main results only use Post?
  3. Could the authors evaluate the method on agentic tasks to demonstrate its practical value for reliability estimation?
Reviewer_NU1GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper estimates the correctness of LLM answers from the hidden states of a frozen model, with two MLP probes. A "competence" probe (Pre) reads the last prompt token at a middle layer and predicts whether at least half of K=30K=30 sampled answers are correct. A "confidence" probe (Post) mean-pools the middle-layer states over the greedy answer and predicts whether that answer is correct. Pre is trained with a class-weighted BCE and Post with a class-weighted focal BCE (γ=1\gamma=1) on their respective hard labels, plus auxiliary BCE terms on the sampled correct rate and on the agreement with the greedy answer (App. A.6). Sampling is used only to build training targets. The deployed score ("SA") is the Post probe alone, so inference needs one generation. Evaluation covers NQ, StrategyQA and MATH on three backbones (plus a Qwen3-32B extension) and zero-shot transfer to TriviaQA, GSM8K, BoolQ and CommonsenseQA, against a post-state probe (KB), a sample-agreement score (CLCM) and token log-probability. Claims: +7.51+7.51 macro AUROC over KB in-distribution (Table 1), 11/12 AUROC wins over KB out of distribution (Table 3), and that Pre adds information beyond Post (App. B).

Critical strengths
  1. Careful, self-critical protocol. No OOD data is used for training, checkpoint selection, thresholds or temperature fitting (App. A.2, A.7). The paper reports three probe seeds, matched controls (Table 21), paired bootstrap CIs (Tables 12, 13, 16) and its own exceptions (Sec. 4.2.1, Table 5). This lets the reader see where the gains come from; most of my concerns are taken from the authors' own tables.

  2. A cheap recipe with a measurable calibration benefit. Training on sampled soft targets but scoring one greedy pass improves over hard-label training on the same features: Brier −0.0155-0.0155 [−0.0165-0.0165, −0.0143-0.0143] and ECE −0.0449-0.0449 [−0.0462-0.0462, −0.0374-0.0374] (Table 16). AUROC with as few as K=10K=10 training samples is statistically indistinguishable from K=30K=30 (Table 23, which reports no Brier/ECE).

Critical weaknesses
  1. The headline gain over KB comes mostly from a stronger probe recipe, not from the proposed ideas.

    • KB uses the same features and target as Post and differs in architecture, loss and optimizer/schedule (App. A.7, A.6), and possibly in checkpoint selection, which is not stated. The paper's own control, a hard-only Post with no auxiliary terms and no Pre stage, already reaches macro AUROC 0.7606 against KB's 0.6886 (Table 21, Table 1). About 7.2 of the 7.51 points come from the recipe. KB is even below chance on Llama2-MATH (0.4862, Table 2).
    • The auxiliary supervision adds only +0.0020+0.0020 AUROC, with a CI that includes zero ([−0.0016-0.0016, 0.00540.0054], Table 16). Pre is not part of the deployed score, and fusing the stages lowers macro AUROC (Table 5).
    • Sec. 4.3 and Table 6 (one seed) say Full is best on all macro metrics, but in the three-seed study (Table 16) Full is slightly worse than Hard+pcorrectp_{correct} on macro Brier (+0.0009+0.0009, CI excludes zero), although its W–L count is 6–3.

    This is decisive because the "two-stage self-awareness" framing is not what the experiments credit for the gain. The evidence supports a narrower claim: soft-label supervision improves calibration of a well-trained single-stage post-answer probe.

  2. The OOD results are not reliable as reported.

    • CLCM has OOD Brier/ECE of 0.76–0.88 on GSM8K while its AURC is not worse than KB's for Qwen3, and chance AUROC on BoolQ and CSQA for Qwen3 (Tables 18–20). In-distribution its Brier is 0.09–0.27 (Table 2). This looks like it may be a failure of answer matching on the OOD formats (see Q2), and it drives the "+11.70+11.70 over CLCM" claim (Table 3).
    • The abstract claims better calibration than the knowledge-boundary baseline OOD. That holds on macro average (8/12 on Brier and on ECE), but on Gemma3 SA is worse than KB on Brier and ECE for all four tasks (Table 20). Untrained log-probability also has lower OOD Brier and ECE than raw SA (0.2056 vs 0.2261; 0.1319 vs 0.1808, Table 17); after ID-fitted temperature scaling SA wins Brier but not ECE (0.1173 vs 0.1068).
    • Transfer is shown only empirically. The OOD sets are close relatives of the ID sets (open QA, math, yes/no and multiple choice), and the paper offers no theory or analysis of why the probe transfers.

    This matters because OOD transfer is one of the two main empirical claims, and as reported it rests on a baseline that may be broken and on a calibration advantage that is not robust across backbones or against untrained log-probability.

Questions for authors
  1. [Clarification] Can you retrain KB with the same architecture, loss, optimizer and checkpoint selection as SA and report it in Tables 1–3? If hard-only Post (0.7606, Table 21) is the right reference, the ID AUROC gain is a few tenths of a point, not 7.51. Please also explain KB's below-chance AUROC on Llama2-MATH.
  2. [Clarification] For CLCM on OOD tasks (Tables 18–20), what fraction of correct greedy answers get agreement 0? Are the K=30K=30 OOD samples generated and normalized exactly like the greedy answer (e.g., the "####" extraction for GSM8K)? A corrected CLCM would change how I weigh the OOD results.
  3. [Clarification] App. A.5 says training and evaluation use cached teacher-forced features (256-token cap, 40.5% of ID answers truncated, App. C.5). Is online inference as in Alg. 1 ever evaluated? Why is Full Post macro AUROC 0.7637 in Tables 1, 5 and 17 but 0.7626 in Tables 21–22?
  4. Two checks on what the Pre signal and the transfer result actually show. [Clarification] (a) The pooled analyses of App. B.2/B.4 (Tables 11–13) mix backbones and tasks with very different base rates. Does Pre still add information with backbone$\times$task fixed effects, or when fit per setting? [Suggestion / future work] (b) To support the transfer claim, could you test a larger shift than near-domain datasets, e.g., train on open QA and test on math?
  5. [Suggestion / future work] Please compare against a temperature-scaled hard-only Post, since Table 17 scales only Full SA and log-probability, and report the OOD calibration of temperature-scaled SA per backbone, given Gemma3.
  6. [Suggestion / future work] The paper offers no mechanistic evidence for why the probes transfer. The layer and pooling controls (App. C.5, Table 22) compare only a few settings, and all main results fix the layer at ℓ∗=⌊L/2⌋\ell^*=\lfloor L/2\rfloor. Could you (a) report a full layer sweep for ID and OOD, to show whether the OOD gain depends on the layer; (b) test whether the probe learns a task-agnostic direction, e.g., by training on one ID task and testing on the other two, or by comparing probes trained on different tasks (e.g., how well one task's probe scores another task's features); and (c) check that the probe is not mainly encoding answer length or format, since Sec. 4.4 uses length diagnostics?
Reviewer_uiS1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes two independently trained MLP probes on frozen LLM representations. Pre predicts whether sampled-answer correctness exceeds a threshold, while Post predicts the correctness of the generated answer. Both receive auxiliary supervision from sampled correctness and answer consistency. Experiments evaluate ranking, calibration, and selective prediction across multiple models and ID/OOD tasks. The primary reported confidence score uses Post alone.

Critical strengths
  • The evaluation covers multiple model families and ID/OOD tasks, using complementary ranking, calibration, and selective-prediction metrics.
  • The matched controls and supervision ablations provide useful evidence about the contributions of answer-state features and auxiliary supervision.
Critical weaknesses
  • Limited methodological novelty. Pre/post-generation internal-state probing, lightweight confidence predictors, and learning from sampled-answer statistics have been explored in prior work. The method primarily combines these established components and assigns different targets to two independent MLPs. The paper does not clearly establish a substantive methodological advance beyond this combination.
  • The central two-stage contribution is not demonstrated. Pre does not contribute to the training or predictions of the primary Post score. In Table 5, neither shared-trunk training nor score fusion improves AUROC over independent Post, although fusion improves calibration. Additional predictive information in pooled diagnostics does not establish that Pre is necessary for the main reported gains. The evidence supports a Post correctness probe more strongly than the proposed two-stage framework.
  • Limited evidence for the full auxiliary-supervision design. The historical KB comparison changes probe architecture and training configuration as well as supervision, so its gains cannot be attributed solely to the proposed supervision. Under matched controls, AUROC increases only from 0.7606 to 0.7626 (Table 21); the paired improvement interval includes zero (Table 16). Hard-plus-soft supervision also outperforms the full objective on the reported macro metrics (Appendix C.1). Calibration improves, but the necessity and additional benefit of the complete supervision design remain unclear.
Questions for authors
  • What substantive methodological advance does the framework provide beyond existing internal-state probes with additional supervision, and which controlled result isolates that advance?
  • What evidence establishes that Pre is necessary for the main claimed improvements, given that the primary score uses Post alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.