acceptodds
Under review as a conference paper at ICLR 2027

Wrong With Conviction: Why Confident Errors Evade Detection in Language Models

Abstract

A language model can be wrong and confident simultaneously, and the confident case matters for deployment: fluent, assertive, and invisible to signals designed to screen low-confidence output. We identify a structural reason for this blindness while showing that the model still retains internally accessible information about whether its answer is correct. We prove an input-invariance theorem for an idealized random-matrix transformer model: when input-dependent gating affects a sublinear number of dimensions and has bounded operator norm, the normalized per-layer transition-Jacobian participation ratio converges to an input-independent constant. Its endpoint-blindness corollary implies that unsupervised macroscopic geometric or spectral readers see the class-dependent part of this geometry diluted by model width below the measurement noise floor, leaving them operationally at chance, with the dilution deepening as models widen. On a nonlinear network, participation-ratio medians differ by 0.87% (AUROC 0.531). Correspondingly, semantic entropy detects genuine uncertainty (AUROC 0.774) but is near chance on adversarial confident error (0.476–0.563 across twelve models and four families). Crucially, blindness is not absence of information: our entity-grouped, out-of-fold commitment-band probe recovers a truth-related direction from frozen last-token residual states, detecting confident error at AUROC 0.731–0.849 across the powered model ladder and retaining AUROC 0.720 [0.674, 0.762] specifically on errors semantic entropy misses. We convert this direction into a Governor that identifies the error regime, routes a regime-matched correction, and refuses when residual risk remains high. Routing is necessary because the wrong regime’s fix actively harms. Across five models and three families, routing beats the best single fix by +4.7 to +10.5 points, with every bootstrap interval excluding zero and McNemar p < 0.001 throughout; routing every item to its regime-matched corrector improves accuracy by +13.0 to +20.4 points over base. Intervention analysis reveals an asymmetric subspace where corruption survives orthogonalization while rescue collapses. Together, these results expose a separation between outward confidence, coarse internal geometry, and recoverable correctness information, showing that confident errors can be diagnosed and acted on rather than treated as terminal failures.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
confident errors, language model reliability, hallucination detection, semantic entropy, uncertainty quantification, internal representations, supervised probing, model steering, selective prediction, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6UqgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors explore whether participation ratio is sensitive to when an LLM is confidently incorrect, and shows that this metric does not show much discriminative power in this setting. The author also shows semantic entropy is a bad predictor of confidence incorrectness, but that a supervised predictor can move AUROC above 0.5.

Critical strengths

None

Critical weaknesses
  • Authors seem to pick a unusual measure of model confidence (participation ratio) and then show the null result on the problem of confident incorrectness prediction. Why should we care about participation ratio in particular? The entire novelty of the paper seems to hinge on a theory based on a statistic that is not very often used in the uncertainty literature, compared to more standard approaches like ensemble variance. This seems to be a paper that has built completely on a straw-man argument.
  • Authors seem to not really understand the space and reinvents concepts. For example "gap" vs "trap" sounds like am imprecise analogue to the breakdown between epistemic vs aleatoric uncertainty, and contextualizing the results around the latter would make the paper a lot more coherent. In general these issues make the paper not feel very connected with pre-existing literature or work in this space.
  • Semantic entropy was trained primarily on accuracy metrics and not confidence metrics, so the inability for semantic entropy to distinguish when an LLM is confidently incorrect is pretty obvious.
  • The other result - that a supervised model trained on supervised confidence signals will outperform 0.5 AUROC - is also pretty obvious.
  • The authors built an entire theory section and then immediately afterwards mention that the primary assumptions of the theory break for LLMs. This seems to invalidate any scientific interest that might exist in the theory.
  • In general, the writing is of poor quality and section titles are difficult to understand, especially insofar as scientific writing goes. Points are disjointed, and there are obvious omissions - for example Figure 1 is one if the main results and is not even referenced in the main text. It feels like large portions of the text, including mathematical theory, were LLM-generated and left as is, with no rewriting from the perspective of scientific rigor or readability. The AI usage statement does reveal that LLMs were used extensively to produce this work, which is why this point is being put here and not being put in the ethics review section, but the amount of apparent reliance on LLMs seems wholly inappropriate for a work of novel scientific scholarship being submitted to a top AI conference.
Questions for authors

(1) Can the authors please contextualize why this work is actually novel with respect to current work? All of the results seem pretty obvious to me.

(2) Can the authors explain why participation ratio in particular matters? If the main result is based on the participation ratio not having discriminative power, this does not prove that any uncertainty statistic would also result in the same lack of discriminativeness, which significantly weakens any scientific value the work may have.

1 comment

Thank you for the review. The paper's contribution is not a null on one statistic plus a classifier above chance. It is (i) a conditional theorem: in an idealized transformer block the normalized Jacobian participation ratio is input-invariant; (ii) the distinction that follows, between error information present in the model and error information visible to a given detector, now stated as a bound that needs no idealization: any score computed from a summary S satisfies |AUROC - 1/2| <= tau - tau^2/2, where tau is the total-variation distance between the class distributions of S (new Proposition 2); (iii) a pre-registered gain of an internal-state probe over the model's own confidence; and (iv) measured consequences for correction. Three statements in the review do not match the paper, and two need clarifying.

  1. Participation ratio. The review says we "pick a unusual measure of model confidence". The participation ratio is not the model's confidence measure or our proposed deployed detector. It is the geometric statistic analyzed in Theorem 1 and checked empirically. The paper separately evaluates model-output confidence and answer margins. The detectors evaluated include semantic entropy on twelve models with a positive control, a dispersion readout, a sparse-autoencoder readout, and a supervised probe. Sections 5 to 9 (regimes, routing, and correction results) are not mentioned in the review's summary.
  2. Gap and trap. These are not our invention and not a version of epistemic versus aleatoric. They follow Simhi et al. (2024, 2025), cited in Sections 1 and 2. Aleatoric uncertainty is irreducible ambiguity in the question; TruthfulQA items have one determinate answer.
  3. Semantic entropy. It is not a classifier trained on accuracy or confidence labels. It computes entropy over meaning clusters of sampled answers (Farquhar et al., 2024); the clustering pipeline uses a fixed, pretrained NLI entailment model.
  4. The probe. It is trained on correctness labels from hidden states, not on confidence signals, and the claim is not "above 0.5". On confident answers the internal state adds to the model's own confidence: +0.090 [0.060, 0.119], pre-registered, positive in 12 of 12 models we had never captured.
  5. The theory. You are right that trained models can depart from the theorem's assumptions; the paper says so. That does not remove the theorem's interest. It is a conditional result, stated as idealized, and the paper measures its conclusion on a trained network, which is standard practice. Re-measured on exact Jacobians the participation ratio is near chance (0.54 [0.49, 0.59]; 0.46 [0.41, 0.52]).

Q2. You are right that a null on the participation ratio does not extend to every statistic. The paper's corollary already limits itself to a function of the participation ratio and says it does not bound arbitrary spectral functionals. We also tested the standard spectral detectors directly on confident errors: EigenScore 0.52 / 0.44 / 0.36 and LLM-Check 0.45 / 0.67 / 0.73 across three model sizes, against 0.88 to 0.95 for a state probe.

Q1, novelty and importance. (a) A measured boundary for semantic entropy: near chance on adversarial items on twelve models (every interval's upper bound below 0.65) against 0.823 [0.734, 0.902] on a general-knowledge control. Farquhar et al. anticipate the direction; the size and the control had not been reported in the cited work. (b) A pre-registered gain of an internal-state probe over the model's own confidence on confident answers, on twelve new models. (c) Direct tests of EigenScore and LLM-Check on confident errors. (d) Matched comparisons of correction strategies: joint and routed correction both beat the best single fix by 5 to 11 points on five models, and the mismatched fix harms. The contribution is not the invention of entropy estimators or probes, but the linked theorem-to-measurement-to-intervention analysis. This matters for deployment: a confident error may evade a coarse score even when other internal information can help detect risk and guide correction or abstention.

Writing. We appreciate the concern about density and presentation. The revision clarifies the theorem's scope, references Figure 1 in Section 3.3, and attributes the participation ratio to Gao et al. (2017). Sections 5 to 9 develop the supervised detector, error regimes, correction experiments, and integrated Governor beyond the participation-ratio result. We hope the revised organization makes these contributions and their limits easier to evaluate.

Language models. ICLR requires an AI-use disclosure section. The submission contains it, as the review notes. The author is responsible for every claim.

Reviewer_hZWLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates an interesting problem regarding why language models can produce confident erros that evade uncertainty-based detection. The overall process includes theoretical analysis of an idealized transformer with experiments on different cases. It argues that internal representations can retain useful error information even when output-based uncertainty measures are ineffective.

Critical strengths
  1. The problem is important as it identifies errors that a language model expresses with confidence. This work examines both uncertainty-based signals and information available in internal representations across multiple models. This comparison helps identify limitations of relying on expressed or sampled uncertainty alone.

  2. The empirical investigation extends beyond detection to the consequences of correction. The paper examines whether different errors benefit from different interventions and reports cases where mismatched corrections are harmful. These results, together with the negative intervention findings, provide useful information about the limits of the proposed approach.

Critical weaknesses
  1. While I appreciate the analysis in Section 3 re the Endpoint-blindness theorem, its connection between the extended discussion needs stronger justification. My understanding is that the theoretical analysis provides narrower conclusions than some other broader claims included in this paper. The formal result concerns a particular statistic under restrictive assumptions, while the discussion extends to semantic entropy and a more general architectural barrier to detecting confident errors.

  2. It is challenging to interpret the error categories and the effectiveness of the complete correction system. The categories are closely associated with question types and data sources, which may not establish whether the model actually possesses the relevant knowledge. Consequently, some benefits of specialized correction could reflect task differences rather than the proposed distinction between error mechanisms.

Questions for authors
  1. How broadly does the theoretical result explain the observed failure of uncertainty-based detection? For example, the formal result concerns a fixed Lipschitz function of normalized participation ratio under an independent noise assumption. What additional argument connects this result to semantic entropy and other spectral detectors? Which parts of this connection are theoretically established, and which are supported only by empirical observations?

  2. How do the authors establish that the proposed error categories represent different underlying knowledge states? Specifically, is there an independent assessment showing that the model possesses the relevant knowledge for “trap” errors but lacks it for “gap” errors? Can the distinction still be identified when question source and format are controlled, so that the router cannot rely primarily on dataset characteristics?

  3. I wonder about the separation of the evaluation of the hidden-state probes from the choices made when developing them. The probe layers were selected using TruthfulQA examples, while TruthfulQA also contributes to the broader evaluation. Were the examples used for layer selection excluded from final testing? Please clarify how layer selection, probe training, and threshold calibration are separated from the reported test results.

  4. Do the correction results establish a benefit from error-specific specialization? How do the two specialized adapters compare with a single adapter trained on their combined data under a comparable training budget? This comparison would help determine whether the gains arise from matching corrections to error regimes or from differences in training data and model capacity.

  5. How effective is the proposed system when detection, routing, correction, and refusal operate together? Are these stages evaluated sequentially on the same held-out examples, including errors introduced by earlier stages? Please distinguish results for the complete system from those obtained by evaluating its components separately, and clarify how the final decision thresholds are chosen.

1 comment

Thank you. Your questions were the right ones, and each now has a measurement behind it.

W1 and Q1, what is proved and what is measured. Proved: input-invariance of the participation ratio for the idealized block. Argued under one further stated assumption: the extension to sampling detectors (Appendix B). Measured: the participation ratio near chance on a trained network (0.531 as submitted; 0.54 [0.49, 0.59] and 0.46 [0.41, 0.52] re-measured on exact Jacobians); semantic entropy near chance on adversarial items across twelve models, with a working control; EigenScore at or below chance on confident errors (0.52 / 0.44 / 0.36 across three sizes). Corollary 1 covers only a function of the Jacobian participation ratio and, as the paper states, does not bound arbitrary spectral functionals; the one sentence that likened it to EigenScore is corrected.

W2 and Q2, error categories and knowledge. The labels are operational, following Simhi et al., and do not certify what a model knows. Measured on 397 errors: 55% re-endorse the misconception, 29% endorse the correct answer when asked, 16% are undetermined. The probe detects both kinds. In our data the trap regime has one source, so regime and source coincide; the revision says so.

Q3, layer selection and evaluation. As Section 5 states, the probe depths were fixed once on 817 TruthfulQA items (about 10.6% of the evaluation pool) and reused unchanged for every model. Two checks address the overlap. First, we re-measured the detection table with every preprocessing step fitted inside training folds and both answer orders on TruthfulQA (which also removes an answer-position artifact we found): 0.65 to 0.79 across eight models. Second, a protocol frozen in advance was run on twelve models never used in development: +0.090 [0.060, 0.119] over the model's own confidence, positive in 12 of 12.

Q4, a joint adapter. We ran it. Both the joint adapter and the router beat the best single specialist on all five models (+5.5 to +10.8 and +4.7 to +10.5 points). Neither beats the other consistently. The gain comes from covering both regimes; applying one regime's fix to the other harms.

Q5, the complete system. As Section 9 states, the submission evaluates correction and refusal separately. Run in sequence on the same 1,013 held-out items (Qwen2.5-1.5B), detector-gated routing reached 0.715 against 0.738 for routing every item. The best measured system corrects every item and then abstains on the corrected model's own margin: accuracy 0.863 at 90% answered (0.921 on Qwen2.5-3B), from base 0.608 (0.750). After correction, the margin is the better abstention signal, and we report that.

Reviewer_B5NkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies confident errors in language models. It presents an input-invariance theorem for the normalized participation ratio of a layer Jacobian in an idealized random-matrix model, which it uses to argue that unsupervised hallucination detectors are blind to confident errors. It then trains probes to detect these errors and a router to select between correctors for “trap” and “gap” failures. The paper argues that handling these regimes separately is necessary for good performance.

The empirical probe results are useful, but I recommend rejection because the connection between the theorem and actual detectors is not established, and the routing experiments lack baselines needed to support the necessity claim. The interpretation of the two failure regimes and missing experimental specifications are additional concerns.

Critical strengths
  • Detection of confident errors across model families: The supervised probes achieve reported AUROC of 0.731–0.849 across the powered model set. This is the paper’s strongest positive contribution: it suggests internal representations can support useful error detection even when the model is confident.
  • Detection on errors missed by semantic entropy: The probe retains reported AUROC of 0.720 on the semantic-entropy-blind subset. This strengthens the practical case for the method by showing that it captures information missed by an uncertainty-based detector, rather than merely duplicating that detector’s signal.
Critical weaknesses
  • The theorem does not establish the proposed explanation for detector failure: Theorem 1 relies on an unverified rank-o(d)o(d) assumption and concerns the participation ratio of a layer Jacobian, rather than the activation-covariance statistics used by the cited spectral detectors. The paper does not establish the connection needed to explain those detectors’ behavior. This is central to my recommendation because the structural explanation is a main claimed contribution, and the empirical detection results do not by themselves validate it.
  • The knowledge-regime interpretation and necessity of routing are not established: Lure-bearing versus ordinary items do not directly measure whether a particular model possesses the relevant knowledge. Moreover, failure of a corrector trained on one regime to generalize to the other does not establish that a single corrector trained jointly on both would fail. Without that baseline, the routing gains support the usefulness of the tested specialists but not the claim that separate correction is necessary. These gaps undermine the paper’s explanation and justification of its proposed correction architecture.
Questions for authors
  1. What connects Theorem 1’s Jacobian participation ratio to the activation-covariance statistics used by EigenScore and LLM-Check? What evidence supports the rank-o(d)o(d) assumption in the trained models?
  2. How do you establish whether a particular model possesses the relevant knowledge, independently of the question’s dataset source and prompt style?
  3. How does the routed correction compare with a single corrector trained jointly on both trap and gap failures, using comparable training data and capacity? Which external correction baselines would provide an appropriate comparison?
  4. Could you specify the retrieval corpus and pipeline for context injection, explain how the three layers’ features are combined in the probe, and provide the two missing gain-over-base entries in Table 3 or explain why they are unavailable?
1 comment

Thank you for a careful review. Each point is answered below, and the revision carries the same changes.

W1 and Q1, the theorem and the named detectors. Theorem 1 is an idealized result, stated as such, with a measured check of its conclusion on a trained network. You are right that EigenScore and LLM-Check read activation covariance, not the Jacobian. Corollary 1 is stated only for a function of the Jacobian participation ratio; the sentence in Section 3.1 that likened it to EigenScore is corrected. We then tested both detectors directly on confident errors (official code, pre-registered, Qwen2.5 0.5B / 1.5B / 3B): EigenScore 0.52 / 0.44 / 0.36, LLM-Check 0.45 / 0.67 / 0.73, state probe 0.88 / 0.95 / 0.89. On generated answers graded by an automated judge the gaps are smaller (1.5B / 3B: secondary forced-choice-state probe 0.71 / 0.73, LLM-Check 0.63 / 0.68, EigenScore 0.56 / 0.48). The primary generation-prompt probe gives 0.66 / 0.76; its paired lead over LLM-Check is unresolved at 1.5B and positive at 3B. The revision also adds a bound that needs no idealization and holds for trained networks as they are: any score computed from a summary S satisfies |AUROC - 1/2| <= tau - tau^2/2, with tau the total-variation distance between the class distributions of S (Proposition 2). The rank condition is a premise of the idealized model. We do not claim trained networks satisfy it, which is why the paper measures the conclusion: re-measured on exact Jacobians, the participation ratio is near chance at both layers (0.54 [0.49, 0.59] and 0.46 [0.41, 0.52]).

W2 and Q2, knowledge states. The trap and ordinary labels follow Simhi et al. and are operational: they describe how an item is built, not a certified knowledge state. We measured the mix. Of 397 TruthfulQA errors on Qwen2.5-1.5B, 55% re-endorse the misconception under verification, 29% endorse the correct answer, and 16% are undetermined. The probe detects both kinds (0.84 [0.77, 0.90] on errors where the model can verify the correct answer; exploratory, 43 errors against 420 correct). The revision states this.

Q3, a joint corrector. We ran it: one rank-8 adapter trained on both regimes at a matched step budget, same 1,013 items. Both trained corrections beat the best single specialist on all five models, by +5.5 to +10.8 points (joint) and +4.7 to +10.5 (routed). Neither beats the other consistently. The Governor's routed design is one working solution, while the jointly trained adapter is a simpler alternative. The final abstract reports both results and does not claim that routing is necessary. Training-free baselines (few-shot, a truthfulness instruction, DoLa, ITI-style steering) scored 0.53 to 0.61 against base 0.608, router 0.738 and joint 0.799 on Qwen2.5-1.5B.

Q4, specifications. Retrieval: live English Wikipedia searched with the question text, up to three article summaries, 500 characters each, prepended to the prompt; text was returned for 145 of 150 errors and 42 were recovered. Probe features: last-token states at relative depths 0.68, 0.79 and 0.86, concatenated. The two missing Table 3 entries are +41.8 and +8.2 points; the range over base across five models is +8.2 to +41.8.

On the strengths you list. We found that the detection table presented the correct option first on TruthfulQA items. Re-measured with both answer orders on TruthfulQA, the detector scores 0.65 to 0.79 across eight models. The claim was then confirmed under pre-registration on twelve models from nine developers that we had never captured: adding the probe to the model's own confidence raises AUROC on confident answers by +0.090 [0.060, 0.119], positive in 12 of 12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.