- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an “AI ontology calibrator” for estimating the trustworthiness of a student LLM’s outputs. A larger expert model provides reference responses offline, and a student token is labeled correct if it exactly matches the expert token at the same position. For each frequent token, the authors project the student’s last hidden state onto a Fisher discriminant direction and set acceptance thresholds from the empirical CDF of the mismatched class. This yields per-token “calibrator probabilities”, which come with DKW-based finite-sample bounds and extensions to α-mixing sequences. These probabilities are aggregated together with softmax and entropy statistics into 15 response-level indicators, and supervised classifiers (logistic regression, MLP) are trained on them to predict response correctness. The evaluation covers BF program synthesis with a custom 22M Transformer, math reasoning with Qwen3-8B and DeepSeek-R1 as expert, and medical MCQ with four small students and GPT-OSS-120B as expert. Labels come from execution, an LLM judge, or ProcessBench-style annotation. Cross-ontology transfer of the classifiers is also reported.
Critical strengths
- Broad and partly verifiable evaluation. The paper covers three heterogeneous domains with five student models, two labeling protocols for Math, and both cross-model and cross-domain transfer. The BF setting is especially valuable because correctness is determined by execution, which gives a clean testbed free of judge noise. This breadth makes the empirical picture informative even where the results are mixed.
Critical weaknesses
- Positioning relative to supervised probes. The paper frames the method as estimating trustworthiness without ground truth. No labels are needed at deployment, but building the calibrator requires expert reference answers, which act as a proxy ground truth, and the response-level classifier is trained on correctness labels for each domain. So the method is therefore close to a supervised correctness probe with an added expert-agreement feature. Since it already relies on student hidden states, the most natural baselines would be probes trained directly on those states (e.g., SAPLMA, INSIDE) and semantic entropy. Without these comparisons, it is hard to tell what the calibrator adds beyond existing probing approaches.
- The token-level label may not reflect correctness. A student token is labeled correct only if it matches the expert token at the same position. Once the two sequences are misaligned by a single token, most later positions would likely be marked wrong. In Math and Medicine, traces from DeepSeek-R1 or GPT-OSS-120B and from Qwen3-8B probably diverge early. The calibrator may mainly encode position within the response rather than the correctness of its content, which would weaken the central premise of the method.
Questions for authors
- For Math and Medicine, could the authors report how the tokens in S₊ are distributed across positions in the response? This would show whether the calibrator captures correctness or mainly prefix position.
- The text states that the proposed indicators consistently outperform traditional measures, but some results seem to suggest a more mixed picture. On Math V1 with ProcessBench labels, the 15 indicators (0.7095) fall slightly below the geometric mean token probability (0.7148), and the gain on Qwen3-0.6B is about 0.02. On its own, p̄ stays near 0.5 on Medicine (Qwen3-8B, Qwen3.5-0.8B) and on Math, and only 25 of 49 transfers favor the calibrator features. Could the authors run the same subset search restricted to traditional indicators, ideally with selection on a held-out validation split?
- Under the same splits, could the authors compare against a probe trained directly on student hidden states to predict response correctness, and against semantic entropy?
If the authors can address the weaknesses and questions above, I would be glad to raise my score.
3 comments
Weakness 2 and Question 1. We agree that the token-level label may not fully reflect correctness. Moreover, token-level correctness is often fundamentally ill-defined, and there is no principled way to decide which individual token is correct or incorrect. Since token correctness is questionable, we evaluate their typically in a given ontology and in a given context relative to an expert whose correctness is assumed or imposed by a design choice. This enables statistical correlation between our indicators and correctness of the downstream tasks when the proportion of analysed tokens is large enough. In order to test this, in experiments we assess these typicality indicators on more interpretable downstream tasks, namely correctness of the entire generation, in several ontologies.
We also agree that if the student generation contains one additional (or one missing) token, all later positions are marked as wrong. Better labeling schemes are possible, for example, treating all student tokens as correct if the whole generation is correct. However, even the simple and computationally cheap approach which we employed in our experiments already yielded good results relative to other comparable methods and approaches available in the literature.
Thank you for the suggestion to examine positional encoding. Figure 9 (provided in the updated manuscript, page 27, middle part) shows histograms of the relative positions of tokens in (token index divided by generation length) for Math and Medicine (MedGemma student). In both cases the distribution is nearly uniform, with no systematic shift toward the beginning, middle, or end of the response. If early divergence and misalignment dominated the labels, would concentrate at the start of the response. Since it does not, the calibrator cannot succeed by simply encoding positions. Nevertheless, even if the bias presented, this is not prohibited by our theory.
We will include this additional ablation and discussion in the final version of the manuscript.
Question 2. Thank you for suggesting this important experiment. We agree that it is necessary to isolate the contribution of calibrator-based indicators from that of traditional confidence indicators.
To address this concern, we conducted an additional ablation study comparing three feature sets: (1) the eight traditional indicators, (2) the five calibrator-based indicators together with the two mixed indicators, and (3) the complete set of indicators used in the submitted paper. For each setting, we report the best-performing feature combination under the experimental selection procedure. This comparison separates the performance of traditional indicators alone from that of calibrator-derived and combined indicators.
Table 6 (provided in the updated manuscript, page 39, middle part) presents the mean and standard deviation of balanced accuracy obtained by an MLP across folds for all seven student/domain settings.
The results show that traditional indicators alone provide strong baselines, while calibrator-based indicators alone generally achieve lower performance. During training, the softmaxes are configured so that the overall generation is correct. Obviously, ours do not have to be correct, because we focus on typicality, not correctness. And when combined, there is a cumulative effect that standard indicators do not show. Where there is ground truth (BF), correctness is linked to typicality, and our metrics are better than the traditional ones.
These results suggest that the calibrator-based indicators provide complementary information to traditional confidence indicators, rather than simply replacing them. The combined feature set consistently achieves higher mean balanced accuracy than the best combination of traditional indicators alone.
We will include this additional ablation and discussion in the final version of the manuscript.
We thank the reviewer for these careful comments. We have added a direct hidden-state baseline, analyzed the positional distribution of the token-level positive class, and conducted additional feature ablations. We address each concern below.
Weakness 1 and Question 3. We thank the reviewer for this point and agree that a direct probe on student hidden states is a natural baseline. Our objective is broader than response-correctness prediction alone: we aim to construct ontology-specific, token-level typicality indicators with finite-sample guarantees, and then test whether their aggregation helps predict response correctness. A direct probe addresses the downstream prediction task, but it does not provide the same token-level statistics or acceptance/rejection bounds. Nevertheless, it is an important complementary baseline, and we have added a SAPLMA-style comparison as described below. We have also revised our positioning to avoid suggesting that the method is entirely unsupervised: the response-level classifier is trained using answer-level correctness labels, while the calibrator uses expert-derived reference labels.
Nevertheless, we have added a SAPLMA-style probe (Azaria & Mitchell, 2023) as a baseline: a supervised classifier that predicts response correctness from the model’s hidden activations at the last answer token. Unlike the original, which uses a small feed-forward network and compares several layers, our probe is a logistic-regression classifier (standardized inputs and class weighting, i.e., the same classifier family as ours) trained on the last-layer hidden state, with fixed hyperparameters and no layer, pooling or model selection. The last-layer state is the input of the LM head, the same representation that our method reads, but our method reads it at every answer token, whereas the probe reads it only at the last one. The probe is trained on the same MedGemma student answers, with the same correctness labels and the same 20-fold stratified cross-validation as our classifier.
We have also added a comparison of indicators computed from the lower bounds on the probabilities of correct acceptance (p) and correct rejection (prej), and from their combination (pcomb). Results are in Table 5 (provided in the updated manuscript, page 38). The SAPLMA-style probe ranks among the top seven methods judged by balanced accuracy. Yet the classifier combining our newly proposed indicators with traditional ones outperforms it. Moreover, unlike our method, SAPLMA provides no guarantees on the probabilities of correct acceptance and rejection. In contrast, our method yields token-level statistics that reflect how typical each generated token is, and their aggregation reflects the correctness of the whole generation. The SAPLMA-style probe, as proposed, reads the activation of a single token and returns one response-level score, so it provides no per-token characteristics.
We will include the results of the SAPLMA-style probe in the final version of the manuscript.
We did not include semantic entropy and INSIDE as main baselines, for a conceptual reason. We distinguish three properties. Uncertainty is a problem-level property that characterizes the spread of the answer space for a given prompt, regardless of any particular output. Consistency measures behavioral stability across multiple outputs and requires at least two generations or inputs to compare. Confidence, by contrast, is an instance-level property of a single prediction: it reflects how strongly internal or expressed signals point to correctness for that specific answer. Our method is a confidence estimator: it scores one specific answer from one forward pass of the student. Semantic entropy estimates uncertainty from the spread of semantic clusters over K sampled answers. INSIDE measures consistency through the dispersion of K sampled-response embeddings. Both therefore target different properties, require K additional generations per query, and do not score the particular answer that is returned. Our token-level baselines and the hidden-state probe are the confidence methods that compete with ours on the same task. We have made this taxonomy explicit in Section 3 (Overall Approach).