acceptodds
Under review as a conference paper at ICLR 2027

Beyond Hidden States: Component-Aware Confidence Estimation for LLMs

Abstract

Large language models can produce fluent but incorrect answers, with confidence signals that often fail to reflect empirical correctness. Existing estimators rely on output-level signals sensitive to decoding and prompts, or probe hidden states as generic high-dimensional features, underutilizing the functional structure of Transformer components. We propose a component-aware confidence estimator that predicts the correctness of a frozen model's answer from three mechanistic descriptors extracted in a single forward pass: Attention Sink Signature, capturing selected heads' sink behavior; Knowledge Response Alignment, measuring directional agreement between attention and feed-forward residual contributions; and Residual Trajectory Geometry, summarizing residual-stream dynamics through normalized distance and angular statistics. A lightweight branch-attentive probe fuses these descriptors against correctness labels with the backbone frozen. Across five question answering benchmarks and four models from the Qwen2.5 and LLaMA-3.1 families, our method achieves the best AUROC and ECE on most combinations and transfers more reliably than internal-state baselines under cross-domain evaluation. These results show that organizing confidence estimation around component-level structure yields more discriminative and transferable signals than treating internal states as a single high-dimensional feature.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Confidence, LLM Probe, Transformer components

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UjEnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper predicts the correctness of an LLM answer using features extracted at the final input-token position before generation. The estimator uses three mechanistic descriptors, selected attention-sink values, cosine alignment between attention and feed-forward contributions, and two normalized residual-trajectory statistics. A small branch-attentive probe is then trained based on these three descriptors. Experiments show the effectiveness of the proposed estimator.

Critical strengths
  • The paper is well-written and easy to follow.
  • The proposed descriptors are reasonably easy to implement for models with internal access.
Critical weaknesses
  • There is no analysis of the statistical properties of the proposed estimator. The current theoretical results establish elementary properties that motivate the feature design, but they do not show the proposed estimator's reliability.
  • The mechanistic justification for the three descriptors is underdeveloped. The current ablations support their predictive usefulness and complementarity, but do not explain why these particular statistics should reflect answer correctness.
Questions for authors
  • What is the purpose of the four elementary properties in Sec. 4.5? I hope the authors explain more about the insights from this theoretical analysis and provide more results on the estimator’s statistical properties.
  • Could the authors design explicit mechanistic hypotheses for each descriptor and provide controlled experiments with statistical tests to evaluate them?
Reviewer_irUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a probe to predict whether a frozen LLM will answer a question correctly, using signals from a single forward pass over the prompt. Instead of raw hidden states, it extracts three compact descriptors: attention-sink values of top-K heads (selected by Cohen's d), per-layer cosine similarity between MHA and FFN updates, and two curvature statistics of the residual-stream trajectory. A small branch-attentive MLP fuses them. Experiments on 5 QA benchmarks with Qwen2.5-7B/72B and Llama-3.1-8B/70B show the best AUROC on 17/20 settings and the best ECE on 17/20, and better cross-domain transfer than I2 and SEP on Qwen2.5-7B.

Critical strengths
  1. The method is cheap. Low-dimensional features from one prompt pass match or beat I2, which uses much larger internal-state features, across two model families and two scales.
  2. Cross-domain results are encouraging: the method beats I2 and SEP on every off-diagonal pair in Fig. 2, and its drop from in-domain to transfer is smaller.
Critical weaknesses
  1. Missing key baselines. Predicting correctness from the question alone is essentially P(IK) (Kadavath et al., 2022), which is cited but not compared. There is also no simple linear probe on the last-token hidden state, and no answer-option probability baseline on the MC datasets. Without these, it is unclear whether the gain comes from the component-aware design. A length confound is also possible: r(x) alone (2 numbers) already reaches ~70 AUROC on GSM8K (Table 2).
  2. Unfair ECE comparison. The proposed probe is trained with BCE on in-distribution labels, so low ECE is expected, while no baseline is recalibrated (e.g., temperature scaling). Some baseline numbers look abnormal: SEP has 61.97 ECE on Llama-70B/MMLU, and EAGLE has 64.08 on Qwen-7B/GSM8K.
  3. Transfer claim is weakly supported. Cross-domain results are only shown on Qwen2.5-7B, several transfer AUROCs in Fig. 2 are below 0.5 (e.g., GSM8K->MMLU 0.41), and most of the advantage comes from transfer among the three MC datasets. Table 1 also has no std, although three seeds are run.
Questions for authors
  1. How does the method compare with a linear probe on the last-token hidden state, P(IK), and the answer-option probability on MC datasets?
  2. Does the advantage remain after controlling for prompt length?
  3. Do the ECE conclusions hold if all methods are temperature-scaled on the same validation split? Please also check the SEP/EAGLE ECE values above 50.
  4. Can you report std over seeds, and cross-domain results on the other backbones?
  5. What is position 0 for Qwen, which does not add a BOS token by default? Are base or instruct models used, and is decoding greedy?
1 comment

Thank you for the rebuttal and the updated manuscript. The added 7B/8B linear probe results, seed std numbers, and Q5 clarifications resolve the setup questions, and I understand the compute limits around 70B/72B inference. Still, four main concerns remain unresolved, most of which need little to no extra LLM forward passes:

  1. On W1/Q1, P(IK) and the MC answer-option probability baseline are still missing on 7B/8B. Since candidate answer logits are already computed for argmax on MC tasks, reporting the option-probability baseline on Qwen2.5-7B and Llama-3.1-8B costs virtually no extra forward passes and is needed to isolate what the internal descriptors add.

  2. On W2/Q3, validation-split temperature scaling only fits one scalar on existing prediction scores with negligible compute, and skipping it leaves calibration uncalibrated. In Table 1, 4 out of 5 baselines, namely Self-Cons., SE, EAGLE, and SEP, are either training-free or trained on semantic entropy rather than 0/1 correctness labels, as confirmed for SEP with 61.97 ECE. Comparing their raw ECE against an in-distribution BCE-trained probe overstates the calibration gain, while against the only same-target baseline I^2, the proposed method's AUROC gains are modest at 0.74 to 2.74 on average and lose on 3/20 settings.

  3. Regarding Q2 and Appendix H.3, the new permutation check only tests concatenated 28-layer full update-direction vectors on MMLU within coarse median-length strata, rather than the proposed 2-scalar descriptor r(x) or the trained probe itself, and it skips GSM8K where r(x) alone hits 70.01 AUROC.

  4. For W3/Q4, cross-domain transfer results are still limited to Qwen2.5-7B, where GSM8K->MMLU drops to 0.41 and most gains come from the three MC datasets. Testing already-trained probes on saved Llama-3.1-8B features requires no extra LLM inference.

Combined with the limitation acknowledged with Reviewer zhHM that the current setting only covers deterministic pre-generation prediction rather than stochastic decoding, I will keep my rating at 2 (Weak rejection) for now, and would be glad to reconsider if these 7B/8B baselines and calibrated comparisons are added.

Reviewer_zhHMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a component-aware confidence estimation framework for frozen large language models. Instead of directly using high-dimensional hidden representations, the method extracts three structured descriptors from Transformer internals:

  1. Attention Sink Signature (ASS): Selects attention heads with high class separability based on their attention mass toward the initial token.
  2. Knowledge Response Alignment (KRA): Measures the cosine similarity between attention and feed-forward network contributions at each layer.
  3. Residual Trajectory Geometry (RTG): Characterizes the evolution of residual representations using normalized displacement and angular statistics. These descriptors are combined using a lightweight branch-attentive classifier trained to predict answer correctness.

Experiments on five question-answering benchmarks and four models from the Qwen2.5 and LLaMA-3.1 families demonstrate strong empirical results. The proposed method achieves the best AUROC and ECE in 17 out of 20 model-benchmark configurations and demonstrates improved cross-domain AUROC relative to two internal-state baselines.

Critical strengths

S1. A useful perspective on structured internal representations.

The paper moves beyond treating hidden states as generic high-dimensional features and instead constructs interpretable, low-dimensional statistics from distinct Transformer components. This is a reasonable and potentially useful research direction.

S2. Extensive evaluation across models and benchmarks.

The evaluation covers five QA datasets, two model families, and model sizes ranging from 7B to 72B parameters. The consistent improvements in the main results are encouraging, particularly against the trained internal-state baseline.

Critical weaknesses

In Section 3 (lines 144–149), the authors define the objective as predicting the correctness of the model's generated answer. However, all three descriptors are extracted from the final input token before generation.

Consequently, the estimator only observes the prompt and the frozen model's pre-generation internal states. It does not observe the answer whose correctness it is supposed to estimate. This creates an important distinction between two different objectives:

  • Prompt-conditioned success probability: How likely is the model to answer a particular question correctly?
  • Answer-conditioned confidence: How likely is a specific generated answer to be correct? For deterministic decoding, these objectives can be closely related because the generated answer is fixed by the prompt. However, with stochastic decoding, the same prompt may produce both correct and incorrect answers, while the proposed confidence score remains unchanged.

Therefore, the estimator cannot distinguish a correct generation from an incorrect generation for the same input prompt. This distinction matters because several compared methods, including Self-Consistency, Semantic Entropy, and output-level confidence methods, use information from generated responses. They may therefore address a different, more answer-specific prediction problem.

Questions for clarity during rebuttal:

  • Explicitly distinguish prompt-level success prediction from answer-level correctness estimation.
  • Evaluate multiple generations for each prompt under stochastic decoding and measure whether the estimator remains calibrated.
  • Include strong prompt-only baselines, such as a hidden-state probe at the final input token, next-token entropy, and simple question-difficulty predictors.
  • Clarify whether the intended application is pre-generation difficulty estimation or post-generation confidence estimation. Without these experiments, the paper's central claim of estimating the confidence of a model's answer is insufficiently qualified.
Questions for authors

Please see above for questions to be considered and discussed during rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.