- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Authors explore whether participation ratio is sensitive to when an LLM is confidently incorrect, and shows that this metric does not show much discriminative power in this setting. The author also shows semantic entropy is a bad predictor of confidence incorrectness, but that a supervised predictor can move AUROC above 0.5.
Critical strengths
None
Critical weaknesses
- Authors seem to pick a unusual measure of model confidence (participation ratio) and then show the null result on the problem of confident incorrectness prediction. Why should we care about participation ratio in particular? The entire novelty of the paper seems to hinge on a theory based on a statistic that is not very often used in the uncertainty literature, compared to more standard approaches like ensemble variance. This seems to be a paper that has built completely on a straw-man argument.
- Authors seem to not really understand the space and reinvents concepts. For example "gap" vs "trap" sounds like am imprecise analogue to the breakdown between epistemic vs aleatoric uncertainty, and contextualizing the results around the latter would make the paper a lot more coherent. In general these issues make the paper not feel very connected with pre-existing literature or work in this space.
- Semantic entropy was trained primarily on accuracy metrics and not confidence metrics, so the inability for semantic entropy to distinguish when an LLM is confidently incorrect is pretty obvious.
- The other result - that a supervised model trained on supervised confidence signals will outperform 0.5 AUROC - is also pretty obvious.
- The authors built an entire theory section and then immediately afterwards mention that the primary assumptions of the theory break for LLMs. This seems to invalidate any scientific interest that might exist in the theory.
- In general, the writing is of poor quality and section titles are difficult to understand, especially insofar as scientific writing goes. Points are disjointed, and there are obvious omissions - for example Figure 1 is one if the main results and is not even referenced in the main text. It feels like large portions of the text, including mathematical theory, were LLM-generated and left as is, with no rewriting from the perspective of scientific rigor or readability. The AI usage statement does reveal that LLMs were used extensively to produce this work, which is why this point is being put here and not being put in the ethics review section, but the amount of apparent reliance on LLMs seems wholly inappropriate for a work of novel scientific scholarship being submitted to a top AI conference.
Questions for authors
(1) Can the authors please contextualize why this work is actually novel with respect to current work? All of the results seem pretty obvious to me.
(2) Can the authors explain why participation ratio in particular matters? If the main result is based on the participation ratio not having discriminative power, this does not prove that any uncertainty statistic would also result in the same lack of discriminativeness, which significantly weakens any scientific value the work may have.
Thank you for the review. The paper's contribution is not a null on one statistic plus a classifier above chance. It is (i) a conditional theorem: in an idealized transformer block the normalized Jacobian participation ratio is input-invariant; (ii) the distinction that follows, between error information present in the model and error information visible to a given detector, now stated as a bound that needs no idealization: any score computed from a summary S satisfies |AUROC - 1/2| <= tau - tau^2/2, where tau is the total-variation distance between the class distributions of S (new Proposition 2); (iii) a pre-registered gain of an internal-state probe over the model's own confidence; and (iv) measured consequences for correction. Three statements in the review do not match the paper, and two need clarifying.
Q2. You are right that a null on the participation ratio does not extend to every statistic. The paper's corollary already limits itself to a function of the participation ratio and says it does not bound arbitrary spectral functionals. We also tested the standard spectral detectors directly on confident errors: EigenScore 0.52 / 0.44 / 0.36 and LLM-Check 0.45 / 0.67 / 0.73 across three model sizes, against 0.88 to 0.95 for a state probe.
Q1, novelty and importance. (a) A measured boundary for semantic entropy: near chance on adversarial items on twelve models (every interval's upper bound below 0.65) against 0.823 [0.734, 0.902] on a general-knowledge control. Farquhar et al. anticipate the direction; the size and the control had not been reported in the cited work. (b) A pre-registered gain of an internal-state probe over the model's own confidence on confident answers, on twelve new models. (c) Direct tests of EigenScore and LLM-Check on confident errors. (d) Matched comparisons of correction strategies: joint and routed correction both beat the best single fix by 5 to 11 points on five models, and the mismatched fix harms. The contribution is not the invention of entropy estimators or probes, but the linked theorem-to-measurement-to-intervention analysis. This matters for deployment: a confident error may evade a coarse score even when other internal information can help detect risk and guide correction or abstention.
Writing. We appreciate the concern about density and presentation. The revision clarifies the theorem's scope, references Figure 1 in Section 3.3, and attributes the participation ratio to Gao et al. (2017). Sections 5 to 9 develop the supervised detector, error regimes, correction experiments, and integrated Governor beyond the participation-ratio result. We hope the revised organization makes these contributions and their limits easier to evaluate.
Language models. ICLR requires an AI-use disclosure section. The submission contains it, as the review notes. The author is responsible for every claim.