acceptodds
Under review as a conference paper at ICLR 2027

A statistical measure of trust in generative AI

Abstract

Trustworthiness is a crucial issue for generative AI, as modern models can produce convincing and confident responses even when their outputs are incomplete, unsupported, or wrong. The frequent absence of ground-truth knowledge or labels complicates verification of generative AI outputs. A further complication is the opacity of training data: in practice, we often cannot distinguish between a model well-trained on low-quality data and a model trained on high-quality data whose internal states nonetheless produce atypical outputs. Trustworthiness can be approached through the lenses of uncertainty, consistency, and confidence – concepts frequently conflated despite their distinct operational meanings. We argue that clearly differentiating them and advancing confidence-based reliability estimation is essential for progress. We propose a trustworthiness framework grounded in model-internal confidence calibrated against an external expert reference. We estimate **per-token trustworthiness** of a student generative model as the statistical typicality of each generated token relative to an expert model's distribution, where the expert is used as a distributional reference rather than as a source of factual verification labels. To this end, we introduce an **AI ontology calibrator** that leverages the student model's internal latent features and token-level output statistics to compute expert-reference acceptance scores. Theoretically, these scores are supported by statistically grounded bounds on threshold-defined acceptance and rejection probabilities under an expert-reference calibration criterion; they should not be interpreted as probabilities of factual correctness. Combined with traditional token-level confidence, entropy, and perplexity statistics, calibrator scores yield a compact **set of 15 response-level indicators** for correctness classification. Evaluated across BF program generation, mathematical problem solving, and medical question answering, combined calibrator and traditional indicators provide complementary reliability signal which could be used to build detectors of untrustworthy outputs and hallucinations. Remarkably, our experiments show that these detectors are transferable across domains, although transferability is not always asymmetric.

Primary area:
foundation or frontier models, including LLMs
Keywords:
trustworthiness, LLMs, AI ontology calibrator

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7oxiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an “AI ontology calibrator” for estimating the trustworthiness of a student LLM’s outputs. A larger expert model provides reference responses offline, and a student token is labeled correct if it exactly matches the expert token at the same position. For each frequent token, the authors project the student’s last hidden state onto a Fisher discriminant direction and set acceptance thresholds from the empirical CDF of the mismatched class. This yields per-token “calibrator probabilities”, which come with DKW-based finite-sample bounds and extensions to α-mixing sequences. These probabilities are aggregated together with softmax and entropy statistics into 15 response-level indicators, and supervised classifiers (logistic regression, MLP) are trained on them to predict response correctness. The evaluation covers BF program synthesis with a custom 22M Transformer, math reasoning with Qwen3-8B and DeepSeek-R1 as expert, and medical MCQ with four small students and GPT-OSS-120B as expert. Labels come from execution, an LLM judge, or ProcessBench-style annotation. Cross-ontology transfer of the classifiers is also reported.

Critical strengths
  • Broad and partly verifiable evaluation. The paper covers three heterogeneous domains with five student models, two labeling protocols for Math, and both cross-model and cross-domain transfer. The BF setting is especially valuable because correctness is determined by execution, which gives a clean testbed free of judge noise. This breadth makes the empirical picture informative even where the results are mixed.
Critical weaknesses
  • Positioning relative to supervised probes. The paper frames the method as estimating trustworthiness without ground truth. No labels are needed at deployment, but building the calibrator requires expert reference answers, which act as a proxy ground truth, and the response-level classifier is trained on correctness labels for each domain. So the method is therefore close to a supervised correctness probe with an added expert-agreement feature. Since it already relies on student hidden states, the most natural baselines would be probes trained directly on those states (e.g., SAPLMA, INSIDE) and semantic entropy. Without these comparisons, it is hard to tell what the calibrator adds beyond existing probing approaches.
  • The token-level label may not reflect correctness. A student token is labeled correct only if it matches the expert token at the same position. Once the two sequences are misaligned by a single token, most later positions would likely be marked wrong. In Math and Medicine, traces from DeepSeek-R1 or GPT-OSS-120B and from Qwen3-8B probably diverge early. The calibrator may mainly encode position within the response rather than the correctness of its content, which would weaken the central premise of the method.
Questions for authors
  • For Math and Medicine, could the authors report how the tokens in S₊ are distributed across positions in the response? This would show whether the calibrator captures correctness or mainly prefix position.
  • The text states that the proposed indicators consistently outperform traditional measures, but some results seem to suggest a more mixed picture. On Math V1 with ProcessBench labels, the 15 indicators (0.7095) fall slightly below the geometric mean token probability (0.7148), and the gain on Qwen3-0.6B is about 0.02. On its own, p̄ stays near 0.5 on Medicine (Qwen3-8B, Qwen3.5-0.8B) and on Math, and only 25 of 49 transfers favor the calibrator features. Could the authors run the same subset search restricted to traditional indicators, ideally with selection on a held-out validation split?
  • Under the same splits, could the authors compare against a probe trained directly on student hidden states to predict response correctness, and against semantic entropy?

If the authors can address the weaknesses and questions above, I would be glad to raise my score.

3 comments

We thank the reviewer for these careful comments. We have added a direct hidden-state baseline, analyzed the positional distribution of the token-level positive class, and conducted additional feature ablations. We address each concern below.

Weakness 1 and Question 3. We thank the reviewer for this point and agree that a direct probe on student hidden states is a natural baseline. Our objective is broader than response-correctness prediction alone: we aim to construct ontology-specific, token-level typicality indicators with finite-sample guarantees, and then test whether their aggregation helps predict response correctness. A direct probe addresses the downstream prediction task, but it does not provide the same token-level statistics or acceptance/rejection bounds. Nevertheless, it is an important complementary baseline, and we have added a SAPLMA-style comparison as described below. We have also revised our positioning to avoid suggesting that the method is entirely unsupervised: the response-level classifier is trained using answer-level correctness labels, while the calibrator uses expert-derived reference labels.

Nevertheless, we have added a SAPLMA-style probe (Azaria & Mitchell, 2023) as a baseline: a supervised classifier that predicts response correctness from the model’s hidden activations at the last answer token. Unlike the original, which uses a small feed-forward network and compares several layers, our probe is a logistic-regression classifier (standardized inputs and class weighting, i.e., the same classifier family as ours) trained on the last-layer hidden state, with fixed hyperparameters and no layer, pooling or model selection. The last-layer state is the input of the LM head, the same representation that our method reads, but our method reads it at every answer token, whereas the probe reads it only at the last one. The probe is trained on the same MedGemma student answers, with the same correctness labels and the same 20-fold stratified cross-validation as our classifier.

We have also added a comparison of indicators computed from the lower bounds on the probabilities of correct acceptance (pp) and correct rejection (prejp_{rej}), and from their combination (pcombp_{comb}). Results are in Table 5 (provided in the updated manuscript, page 38). The SAPLMA-style probe ranks among the top seven methods judged by balanced accuracy. Yet the classifier combining our newly proposed indicators with traditional ones outperforms it. Moreover, unlike our method, SAPLMA provides no guarantees on the probabilities of correct acceptance and rejection. In contrast, our method yields token-level statistics that reflect how typical each generated token is, and their aggregation reflects the correctness of the whole generation. The SAPLMA-style probe, as proposed, reads the activation of a single token and returns one response-level score, so it provides no per-token characteristics.

We will include the results of the SAPLMA-style probe in the final version of the manuscript.

We did not include semantic entropy and INSIDE as main baselines, for a conceptual reason. We distinguish three properties. Uncertainty is a problem-level property that characterizes the spread of the answer space for a given prompt, regardless of any particular output. Consistency measures behavioral stability across multiple outputs and requires at least two generations or inputs to compare. Confidence, by contrast, is an instance-level property of a single prediction: it reflects how strongly internal or expressed signals point to correctness for that specific answer. Our method is a confidence estimator: it scores one specific answer from one forward pass of the student. Semantic entropy estimates uncertainty from the spread of semantic clusters over KK sampled answers. INSIDE measures consistency through the dispersion of KK sampled-response embeddings. Both therefore target different properties, require KK additional generations per query, and do not score the particular answer that is returned. Our token-level baselines and the hidden-state probe are the confidence methods that compete with ours on the same task. We have made this taxonomy explicit in Section 3 (Overall Approach).

Weakness 2 and Question 1. We agree that the token-level label may not fully reflect correctness. Moreover, token-level correctness is often fundamentally ill-defined, and there is no principled way to decide which individual token is correct or incorrect. Since token correctness is questionable, we evaluate their typically in a given ontology and in a given context relative to an expert whose correctness is assumed or imposed by a design choice. This enables statistical correlation between our indicators and correctness of the downstream tasks when the proportion of analysed tokens is large enough. In order to test this, in experiments we assess these typicality indicators on more interpretable downstream tasks, namely correctness of the entire generation, in several ontologies.

We also agree that if the student generation contains one additional (or one missing) token, all later positions are marked as wrong. Better labeling schemes are possible, for example, treating all student tokens as correct if the whole generation is correct. However, even the simple and computationally cheap approach which we employed in our experiments already yielded good results relative to other comparable methods and approaches available in the literature.

Thank you for the suggestion to examine positional encoding. Figure 9 (provided in the updated manuscript, page 27, middle part) shows histograms of the relative positions of tokens in S+S_{+} (token index divided by generation length) for Math and Medicine (MedGemma student). In both cases the distribution is nearly uniform, with no systematic shift toward the beginning, middle, or end of the response. If early divergence and misalignment dominated the labels, S+S_{+} would concentrate at the start of the response. Since it does not, the calibrator cannot succeed by simply encoding positions. Nevertheless, even if the bias presented, this is not prohibited by our theory.

We will include this additional ablation and discussion in the final version of the manuscript.

Question 2. Thank you for suggesting this important experiment. We agree that it is necessary to isolate the contribution of calibrator-based indicators from that of traditional confidence indicators.

To address this concern, we conducted an additional ablation study comparing three feature sets: (1) the eight traditional indicators, (2) the five calibrator-based indicators together with the two mixed indicators, and (3) the complete set of 1515 indicators used in the submitted paper. For each setting, we report the best-performing feature combination under the experimental selection procedure. This comparison separates the performance of traditional indicators alone from that of calibrator-derived and combined indicators.

Table 6 (provided in the updated manuscript, page 39, middle part) presents the mean and standard deviation of balanced accuracy obtained by an MLP across 2020 folds for all seven student/domain settings.

The results show that traditional indicators alone provide strong baselines, while calibrator-based indicators alone generally achieve lower performance. During training, the softmaxes are configured so that the overall generation is correct. Obviously, ours do not have to be correct, because we focus on typicality, not correctness. And when combined, there is a cumulative effect that standard indicators do not show. Where there is ground truth (BF), correctness is linked to typicality, and our metrics are better than the traditional ones.

These results suggest that the calibrator-based indicators provide complementary information to traditional confidence indicators, rather than simply replacing them. The combined feature set consistently achieves higher mean balanced accuracy than the best combination of traditional indicators alone.

We will include this additional ablation and discussion in the final version of the manuscript.

Reviewer_2MKTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a new approach to evaluating "trust in generative AI" through the lens of confidence. The main idea is to assess the trustworthiness of a target model's generation by comparing its internal token-level confidence against the distribution of a large teacher (expert) model. In this paper, the expert model is viewed as a distributional reference rather than a source of factual verification labels.

Critical strengths

The paper has several positive aspects:

  • First, evaluating the trustworthiness of generative AI is a critical topic.
  • Second, framing the proposed method as a complementary reliability signal, rather than a strict fact-checker, is a sensible and well-motivated idea.
  • Third, the paper approaches the target problem from a practical standpoint (e.g., extracting lightweight indicators and training a small classifier so that the method can be deployed more practically).
Critical weaknesses

However, the following aspects need to be improved or addressed:

  • Although the paper mentions that the teacher serves as a distributional reference rather than a source of factual labels, the teacher's distribution itself may still contain incorrect information or biases. While the references cited in the paper (Yuan et al., 2021; Fu et al., 2024) suggest that the distributions of recent large models correlate with human judgments, this does not necessarily prove that existing models are reliable enough to serve as reference sources. For example, if the large teacher model holds a misunderstanding, misrepresented concept, or bias, evaluating the target model's distributional trustworthiness against it could create a negative feedback loop. Thus, some form of human guardrails or verification may still be needed to resolve this issue.
  • Furthermore, the paper proposes training a small calibration model rather than querying the large teacher model every time. While the concept itself is reasonable, there is insufficient supporting evidence regarding the quality of the small calibration model (i.e., whether the calibrator effectively captures the original teacher model). Although the paper provides downstream experiments to show the effectiveness of the calibrators, it would be very helpful to see a more direct comparison between the calibrator's output and the teacher's output.
Questions for authors

The questions are included in the "Weaknesses" section above.

1 comment

We thank the reviewer for these thoughtful comments. We address the two concerns separately: the possibility that the expert reference contains errors or biases, and the interpretation and evaluation of the calibrator itself.

We first clarify the distinction between the expert-derived token-level signal and the final response-level correctness prediction.

Weakness 1. Thank you for this comment. However, there is a misunderstanding. Calibrators are not used for predictions. They have two functions: quantifiable and measurable representation of probability of individual token generation in the given context. Softmax scores although related have never been intended to service to genuine calibrator of probability of the correctness. Their function is to model distributions for sampling next tokens along the model's trajectory. In our work, we introduce an additional measure (score) of the probability of token correctness relative to the expert in the given context. This probability is only loosely related to the correctness of the answer and therefore the calibrators cannot be used for judging the quality of the final answer on their own. Nevertheless, producing estimates to correct answers we introduce a family of classifiers (in the paper presented as Logistic Regression and MLP) taking classical indicators and the newly produced calibrators’ indicators as inputs and outputting the probability of the whole generation correctness.

In the revision, we will enhance our current text and the presentation by explicitly describing this process in the Introduction, Abstract, and Conclusion so that the overall approach is presented clearer.

Weakness 2. As we also highlighted for Reviewer doRv in answer W2, the token correctness is the philosophical question. Moreover, our calibrator provides per-token probabilities rather than the value we can directly test. In cases where the truthfulness is explicitly defined and the expert is trustful as in the BF case (because we have the oracle – BF programs compiler), we have statistically significant improvement after adding calibrators’ indicators to the traditional ones (also provided the results in the answer to Reviewer doRv, W1). In more complex domains such as medicine, the correctness of answers is more complex and the expert is not an ideal expert, as a result, the improvement is statistically significant only for the case where the fraction of calibrated tokens in answers is higher than 0.850.85. Nevertheless, our method provides more than a single confidence or hallucination score. For each generated token, it estimates two complementary probabilities: the probability of correct acceptance and the probability of correct rejection relative to the expert-derived reference distribution. These estimates provide token-level statistical information about how typical the generated token is under the reference ontology, together with finite-sample guarantees. Aggregating these token-level indicators allows the downstream classifier to assess the correctness of the entire generation.

Reviewer_doRvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an AI ontology calibrator that estimates trustworthiness of a student generative model using an expert model as a reference. Their framework measures the statistical confidence of each token relative to the expert (offline) and determining whether the student generated response is trustworthy. They do this by using the student models statistics on a generated token via calibrator probabilities, softmaxes, and entropies. They derive a set of indicators from these quantities. They then feed these into a classifier to make a judgment on whether the response is correct. The authors provide experiments over three domains; BF, Math, and Medicine. They mostly outperform individual baselines in most settings with reported balance accuracies around 0.65-0.85.

Critical strengths
  1. This is an important well posed problems. Cheaply training a classifier to help assess the quality of a student model via an expert model offline is also a very useful and efficient way to tackle an important problem.
  2. The experiments are comprehensive with respect to domains. The experiments focus on several domains and compare against numerous single feature baselines.
Critical weaknesses
  1. Under the current framework the calibrator treats agreement with the expert as trustworthiness so a student matching an incorrect expert would be scored as trustworthy. The authors do state that the expert is not an oracle however this doesn't solve the issue of a calibrator simply parroting an incorrect expert.

  2. The experiments suggest only a minor contribution or at least they do not demonstrate a clear contribution from the calibrator. pˉ\bar{p} is almost at a coin flip for four medicine students. Also the contribution from the calibrator is never isolated the full 15 indicator set also contains 8 traditional features so it could be seeing gains from traditional features. For example on Medicine Qwen3-0.6B: LM PP alone achieves 0.822 balanced accuracy vs 0.843 for all 15 indicators

Questions for authors

The work is important and i would be willing to change my score if the authors can address my concerns:

  1. How does the framework handle cases where the expert is incorrect given that the calibrator scores agreement with the expert regardless of correctness? Providing experts accuracy on each domain and calibrator's behaviour on items where the expert is wrong would be useful.
  2. The experimental results do not isolate the calibrators contribution. How can the authors assert that the gains are not simply due to adding traditional features? It is better to report the same classifier trained on traditional features only vs traditional features plus calibrator features.
1 comment

We thank the reviewer for these thoughtful and constructive comments. We have conducted additional analyses to clarify the role of the expert, examine the contribution of the calibrator-based indicators, and better characterize the limitations of the proposed approach. We address each point below. We hope these responses eliminate concerns raised by the reviewer.

We thought that these critical points were handled in the manuscript albeit in hindsight we agree that this could be done better. There are two weaknesses highlighted:

  1. Experiments do not show the impact of new indicators.
  2. Conditioning of calibrators' performance on the truthfulness of experts.

Weakness 1 and Question 1. The additional experiments address the issue from two complementary perspectives: an automatically verifiable setting with oracle (BF), where program correctness is established by execution, and a medical setting, where the expert’s references are used which may itself be imperfect.

For both cases, we additionally conducted the statistical tests to check whether adding the calibrator-based indicators significantly improves correctness classification. For this, we ran a one-tailed paired t-test with H1: Δ=mean(dj)>0\Delta = \text{mean}(d_j) > 0, where jj is the fold (from 11 to 2020) and dj=BAj(traditional + calibrator indicators)−BAj(traditional)d_j = BA_j(\text{traditional + calibrator indicators}) − BA_j(\text{traditional}), BABA denotes balanced accuracy. The test parts of the folds do not overlap, but the training sets do. We therefore also applied the Nadeau-Bengio corrected t-test, and Table 4 (provided in the updated manuscript, page 33, middle part) reports these results for BF ontology. We adjusted p-values with the Holm method, separately for the two comparison groups (traditional indicators with and without geometric means). A result is significant if Δ>0\Delta > 0 and p(Holm)<0.05p(\text{Holm}) < 0.05.

In addition to the probability of acceptance presented in the manuscript, we included the probability of correct rejection – for which our method also provides guarantees – as well as their combined version.

For Medicine, the effect is more dependent on how many tokens in an answer can be calibrated. Some questions and answers contain rare, domain-specific terms, whereas we build calibrators only for tokens observed at least 1,0001,000 times. Figure 14 (provided in the updated manuscript, page 34, top part) shows the distribution of the fraction of calibrated tokens per answer for MedGemma. Although the largest group of answers has 70–8070–80% calibrated tokens, a non-trivial fraction has lower coverage. We therefore performed an additional analysis in which, for each training fold, we retained only training examples whose calibrated-token fraction exceeded a threshold THTH. We considered THTH from 0.400.40 to 0.850.85 in increments of 0.050.05, as well as no filtering (TH=0TH = 0). This analysis is intended to characterize how coverage affects the usefulness of the indicators.

In Figure 15 (provided in the updated manuscript, page 35, top part), stars indicate statistically significant improvements for the comparisons shown. For MedGemma, the gains become statistically significant at the higher calibrated-token coverage threshold (TH=0.85TH = 0.85) for the indicated comparisons. This result suggests that the contribution of calibrator-based indicators depends on coverage and should be interpreted conditionally, rather than as a uniform improvement across all medical examples.

We will include these additional analyses and discuss the dependence on calibrated-token coverage in the final version of the manuscript.

Weakness 2 and Question 2. This weakness point is more philosophical rather than technical since it touches upon the origin and nature of truthfulness. We agree that the expert's knowledge may contain errors or biases. Our method does not establish truth by definition of agreement with the expert, nor can it guarantee that an expert-derived reference is factually correct. The expert provides a distributional reference used to construct token-level typicality indicators; the final response-level classifier is trained and evaluated against separate answer-level correctness labels. Consequently, expert errors can weaken or bias the indicators, and the method should not be presented as a substitute for human or domain-specific verification. We will clarify this limitation and describe the method as an ontology-relative, complementary signal rather than an independent source of ground truth.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.