acceptodds
Under review as a conference paper at ICLR 2027

Measuring Internal Uncertainty in Language Model Agents: A Multi-Method Triangulation Approach

Abstract

Adaptive information acquisition requires an agent to assess uncertainty about its current judgement. For large language models (LLMs), however, it remains unclear which observable signals provide useful measures of this internal uncertainty. We use a controlled probabilistic reasoning task with evidence that is either ambiguous, strongly supports the correct answer, or strongly supports an incorrect answer. This design allows us to test whether candidate uncertainty measures respond to how conclusive the evidence is, even when it is misleading. We compare candidate measures derived from verbal reports, answer-option distributions, opt-out behaviour, reasoning traces, and internal representations, measured in GPT-OSS-20B and Qwen3.5-27B. We evaluate the ability of these candidate indicators to distinguish evidence conditions, retain a set of effective indicators, and assess their convergence across measurement methods. We construct two composite indices that consistently distinguish evidence conditions in both models. In Qwen3.5-27B, higher measured uncertainty is further associated with sampling beyond the model's currently preferred cue. These findings provide a measurement basis for investigating how internal uncertainty relates to information acquisition and decision-making.

Primary area:
interpretability and explainable AI
Keywords:
metacognition, decision-making, information seeking, interpretability, machine cognition, belief elicitation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3HYHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tests methods to measure LLM uncertainty using ambiguous and misleading cues. They test verbal reports, answer probabilities, opt-out behavior, reasoning traces, and hidden-state measures in GPT-OSS-20B and Qwen3.5-27B. Several indicators distinguish ambiguous from concentrated evidence; their effectiveness depends on the model and reasoning setting. However, the work is mainly a measurement study; it does not establish a new method or directly contribute to internal belief-state interpretability.

Critical strengths

A key strength is the controlled contrast between ambiguous and confidently misleading evidence, which addresses the common conflation of uncertainty with correctness. The multi-method comparison on shared histories also provides a useful map of which observable indicators converge and which depend on the model or reasoning setting.

Critical weaknesses
  • The paper doesn't have clear definitions of the problem discussed. Notions are used without clear definitions. It is very difficult to follow when reading the text. For example, do XtX_t and kk both indicate the cue? It is very confusing to mix two types of notions. I think the paper needs to clearly discuss the problem setup and motivate the experimental design.

  • The contribution is limited; What I understand of the paper is that the strongest idea is the misleading-cue-controlled experiments. The model that can be confident but wrong is well known. But the paper's contribution didn't provide a novel method to mitigate the problem.

  • The claim of an internal uncertainty measure is quite misleading; the line of work to study internal uncertainty aims to decode the model's internal representation to estimate uncertainty. Although the paper uses a trained probe and Jacobian lens, the contribution to internal uncertainty interpretability is limited.

Questions for authors

It seems that you sampled a cue from the 15 trials and observed the probability of the outcome as RED, and the probability is then used to measure M1 and M2. How do the numbers of observations per cue differ across the veridical, ambiguous, and misleading conditions? Could the authors report these distributions and repeat the main indicator comparisons with balanced cue sampling, or with a minimum number of observations per cue?

2 comments
AuthorsOpenReview ↗

Author response (1/2)

We thank the reviewer for their comments and questions, which we address below. We believe that some of these may arise from a slight misunderstanding of our research objectives and research contributions. We hope that our responses clarify these points and we will also address them in the revised version of the paper. Given these clarifications, we feel that the negative ratings may be harsher than warranted, and hope the reviewer will reconsider them.


Weakness 1: "lack of clarity"

  • To clarify, XtX_t and kk refer to the same cue set, C=A,B,C,D\mathcal{C}={A,B,C,D}, but serve different roles. XtX_t, defined in Section 3.1, denotes the cue selected by the agent in round tt. By contrast, kk in Section 3.2 is an index ranging over the candidate cues, used in quantities such as pt,k=P(Z=k∣ht)p_{t,k}=P(Z=k\mid h_t). Thus, XtX_t specifies the agent’s actual selection, whereas kk indexes the hypotheses being evaluated. For example, selecting cue A (Xt=A)(X_t=A) does not preclude evaluating the posterior probability that cue B is biased (pt,B)(p_{t,B}).
  • The task setup and notation are introduced in Sections 3.1–3.2. We will revise these sections to make the definitions and motivation generally clearer and more explicit. Beyond the distinction between XtX_t and kk, could the reviewer specify which other notions or parts of the problem setup they found unclear? This would help us make more targeted improvements to the presentation, particularly given that lack of clarity was not flagged by the other reviewers.

Weakness 2: "no new method"

We thank the reviewer for the comments on this, but we are concerned they may arise from a slight misunderstanding of the research objectives of our paper and perhaps also from how explicitly we presented our findings. We will try to clarify.

Our core question is a measurement question: which observable signals can serve as valid indicators of internal uncertainty; do different measurement approaches give consistent estimates; and how do estimates of internal uncertainty relate to behaviour? Demonstrating or mitigating confident errors are related research avenues, both interesting and valuable, but they are not what our paper is about. Similarly, while our study is motivated by the (known) fact that models can be confident but wrong, that is not, nor do we want to claim this known fact as our discovery.

We do want to underline that, although we do not propose a mitigation method, the measurement validation protocol is itself a methodological contribution. We propose a procedure grounded in psychometric principles (1) to validate uncertainty indicators (some of which were never used in this way before) without using answer accuracy as the validation criterion—a procedure that applies to both behavioural, output-level and representation-level signals—and (2) to aggregate the validated indicators into composite indices.

As for the comment on misleading-cue-controlled experiments: this is one of the empirical findings we obtain by applying the measurement instrument validation procedure described above—other findings being, e.g., that only a subset of existing candidate measures satisfy the validation criteria, and that some indicators are sensitive to the specific language model and reasoning setting. These empirical findings are a contribution of the paper as well.

In particular, we would like to highlight the finding that, after removing condition-level mean differences, the representation-based indicators we tested do not show reliable convergence with output-based indicators and exhibit weak associations with subsequent sampling behaviour. We believe this has direct implications for interpretability research and method development.

Overall, we take the criticism that the objectives and contributions of the paper, including the impact on interpretability research, could be stated more explicitly; we will do so in the introduction of the revised paper.


AuthorsOpenReview ↗

Author response (2/2)

Weakness 3: "no internal uncertainty interpretability"

As stated in Section 2, "internal uncertainty" refers to the model’s assessment, based on the information acquired so far, of the extent to which that information supports a stable representation of an unknown state of the world. "Internal" emphasises that this assessment is generated within the model, that it cannot be directly observed, and that we therefore treat it as a latent construct. This does not mean that it must be measured by decoding internal model representations (just like measuring a person’s uncertainty doesn’t require decoding their brain activity).

That said, decoding uncertainty from internal representations is one of the measurement approaches considered in our paper, alongside verbal reports, abstention behaviour, output distributions, and reasoning traces. Some of the specific probe-based, Jacobian lens-based, and SAE-based methods used to measure internal uncertainty are, to our knowledge, novel, and (though this is irrelevant to the objective assessment of the contribution) we had high hopes for them!

Our negative empirical results on using these interpretability methods are also new and, as discussed in the previous answer, they do inform work on internal representations. To recap, applying our measurement validation criteria to probe-, Jacobian-lens and SAE readouts, we found that some fail validation altogether (Jacobian lens; the Qwen3.5-27B SAE score on held-out data) and that those that pass do not converge with output-based indicators within conditions (r ≤ .16) and do not predict subsequent sampling (see 1.2). We believe this is a valuable finding for researchers working on "internal uncertainty interpretability". We will mention these contributions more explicitly in the introduction.


Question: distribution of observations

Cue sampling in our design is balanced. The number of observations per cue is nearly identical across the three conditions. Each cue is observed about 15/4 ≈ 3.75 times on average across conditions:

ConditionABCD
Veridical3.783.703.783.74
Ambiguous3.863.673.653.82
Misleading3.843.683.743.74
Reviewer_GaGUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates uncertainty indicators in GPT-OSS-20B and Qwen3.5-27B using a synthetic four-cue probabilistic reasoning task. An ideal Bayesian observer defines ambiguous, veridical, and misleading evidence conditions. Authors compare verbal reports, answer entropy, opt-out behavior, reasoning-trace measures, and representation-based readouts. Selected indicators are combined into two composite scores. Authors validate using 300 fixed histories, intermediate-round measurements, and 30 active trials with Qwen. The composites distinguish the selected evidence conditions and are associated with subsequent sampling choices in the active task. The motivation and task are clearly explained. The appendices provide substantial detail, although the data separation behind the held-out composite indicators needs clarification.

Critical strengths

The paper separates uncertainty from answer correctness. Strong but misleading evidence can support a confident incorrect answer. Including this condition makes the evaluation more informative than assessing indicators only through correctness prediction. The comparison covers several measurement sources and reports failures. Behavioral, output-based, and representation-based indicators are evaluated within a common task. Setting-dependent results and the failed Qwen SAE confirmation help identify the limitations of individual measures

Critical weaknesses
  1. The benchmark does not sufficiently distinguish internal uncertainty from sensitivity to observable evidence statistics. The evidence conditions are defined by the ideal observer’s posterior. Appendix A shows that this posterior is a deterministic function of cue-specific outcome counts: s_k=r_k\log 1.8+g_k\log 0.2p_k=\operatorname{softmax}(s)_k. A representation readout that separates these conditions could encode counts, likelihood ratios, or posterior concentration. I dont see sufficient evidence distinguishing these explanations from the proposed internal uncertainty assessment. This is a construct-validation issue. Campbell and Fiske [1] distinguish common construct variance from measurement-method variance. Fleming and Lau [2] distinguish metacognitive bias, sensitivity, and efficiency. The paper should position its proposed motication against these established distinctions and explain which interpretation its evidence supports. This does not require treating correctness as uncertainty ground truth. The probe results require similar case. Hewitt and Liang [3] explain why successful probing needs control before supporting conclusions about represented information. The authors’ condition-preserving permutations are a relevant control, but they do not directly separate continuous evidence-statistic encoding from uncertainty-assessment encoding. The evaluation population is also restricted. Approximately 78.5% of the 60,000 generated histories satisfy none of the three condition definitions. Ambiguous and misleading histories each account for less than 0.7% of generated trials, but each receives one-third of the evaluation weight. This is reasonable for a controlled contrast. However, the AP values describe that selected population, and precision–recall performance depends [6]. The intermediate-round analysis still uses prefixes of endpoint-selected histories; balancing concentration bins does not remove this conditioning. Finally, most validation uses retrospectively supplied histories. The active experiment contains only 30 trials with one model. The agent framing is defensible for that experiment, but the scope of the conclusions should reflect how much evidence comes from passive reasoning.

  2. The screening and evaluation procedures provide weaker validation than the central interpretation requires. The main screening rule retains indicators when the ambiguity contrasts are significant and the veridical–misleading contrast is nonsignificant. However, p_{V-M}>0.05 \quad\not\Rightarrow\quad |\mu_V-\mu_M|<\delta. As discussed by Lakens [4], failing to detect a difference does not establish equivalence. The paper acknowledges this distinction, but its final screening rule still relies on nonsignificance. Convergence is then assessed through correlations pooled across the conditions used for screening. Shared condition differences can produce positive correlations even when indicators have no relation within conditions. These correlations support common sensitivity to the manipulation, but provide limited evidence of shared trial-level variation. Condition-adjusted results would make the interpretation clearer. Evaluation independence also needs clarification. Appendix D.6 states that probe layer selection and performance reporting use the same cross-validation results. Cawley and Talbot [5] describe how optimizing a finite-sample validation criterion can produce optimistic estimates. The permutation procedure also does not repeat layer selection or adjust for the search, so its p-values do not account to selecting the reported configuration. The authors appropriately call these estimates exploratory; the downstream interpretation needs the same qualification. There is also a specific issue with Equation 28: R= \frac{\min(|d_{\text{correct}}|,|d_{text{incorrect}}|)} {|d_{\text{all}}|}. Because this discards effect direction, it can pass its threshold even when the contrast reverses in one correctness subgroup. Unless an additional sign constraint is imposed, a large retention ratio does not establish preservation of the effect across correctness

Questions for authors

hese questions mainly request clarification or checks using existing outputs, rather than a new experiments.

  1. Construct interpretation . What reported result distinguishes internal uncertainty from encoding cue statistics or ideal-observer posterior concentration? How does the proposed method relate to established distinctions between metacognitive bias, sensitivity, and efficiency?
  2. Equivalence . What magnitude of veridical–misleading difference would be practically unacceptable? Which retained indicators exclude that magnitude through confidence intervals or equivalence tests, rather than merely yielding a nonsignificant difference?
  3. How much association remains within evidence conditions, or after adjusting for condition? A small table from existing indicator scores would clarify whether convergence extends beyond the shared group differences.
  4. Selection and evaluation . Please identify the trials or folds used for indicator selection, reasoning-setting selection, probe layer selection, threshold fitting, standardization, and final evaluation. What exactly are the “held-out flags”? How should the probe significance results be interpreted given that permutations do not repeat layer selection?
  5. Are the AP claims restricted to the balanced, selected population? Which conclusions are supported for the remaining generated histories? Please also clarify how endpoint selection affects interpretation of the intermediate-round results.
  6. Does Equation 28 additionally require correctness effects to have the same sign as the overall effect? Please provide the signed subgroup effects for the retained SAE scores.
  7. Is preferred-cue resampling evaluated only when that cue is available? Please confirm that scores through round (t) are paired with choices at (t+1), and explain how posterior concentration, round, and repeated observations within trials are handled.
7 comments
AuthorsOpenReview ↗

Author response (1/6)

We thank the reviewer for the careful and constructive review. The questions led us to run several new analyses, which we report below. In brief: a test with identical input shows that some output-level indicators carry information beyond the evidence statistics; a new control shows that the probes do not; and we now report equivalence tests, condition-adjusted correlations, permutation tests that repeat the layer search, and the active experiment with a second model. Several of these results narrow our claims, and we say where. We hope the reviewer will find the revised claims well supported and will reconsider their rating.

Weakness 1 and Question 1: "construct interpretation"

We thank the reviewer for raising this important point. We agree that, in the original analysis, a readout that separates the evidence conditions could encode cue counts, likelihood ratios, or ideal observer posterior concentration rather than the model's internal uncertainty. We conducted additional analyses to verify this.

Analysis 1. In a first analysis, we held all observable evidence statistics constant. We gave each model the same history ten times. Within each set of repetitions, the input, cue counts, likelihood ratios, and ideal observer posterior are identical. These quantities therefore cannot explain any variation between the ten responses.

The histories are the 3,720 round-by-round histories (rounds 1–15 of the 300 trials) in which a single cue has the highest ideal observer posterior. We retained those for which the model did not produce ten identical responses: 1,455 histories (39%) for GPT-OSS-20B at low reasoning effort and 271 (7%) for Qwen3.5-27B. For each sampled response, we recorded whether the model selected the cue with the highest ideal observer posterior. We then tested whether an uncertainty indicator was higher when the model did not select that cue. If an indicator merely encoded cue counts, likelihood ratios, or posterior concentration, it could not discriminate between responses within the same history, because all these quantities remain fixed. Above-chance discrimination would instead show that the indicator encodes information about the model's processing or response selection in specific samples. We quantify this using AUROC, with chance level 0.5. The results were as follows:

  • GPT-OSS-20B, low reasoning effort: answer entropy had an AUROC of 0.67 (95% CI [0.63, 0.71]), and the verbal uncertainty report had an AUROC of 0.59 [0.57, 0.62].
  • Qwen3.5-27B: reasoning length had an AUROC of 0.58 [0.53, 0.62], and uncertainty-word use had an AUROC of 0.60 [0.56, 0.63].

Every confidence interval excludes chance level; therefore, for both models, at least some indicators contain information that is not determined by the observable evidence statistics or ideal observer posterior. For GPT-OSS-20B at low reasoning effort, these indicators are answer entropy and the verbal uncertainty report. For Qwen3.5-27B, they are reasoning length and uncertainty-word use. The other indicators tested do not show this pattern (AUROC between 0.20 and 0.53), so we make this claim only for these four; which indicator carries the signal differs by model and reasoning setting, in line with the main conclusion of the paper.

Analysis 2. We conducted a second analysis where we did not hold the evidence statistics exactly constant but rather compared different histories that have similar evidence (same condition, same M2 bin). We tested how well each indicator tells apart the answers where the model chose the best-supported cue from the answers where it did not (AUROC). We compared this with a simple rule that uses only the counts: is the cue with the highest share of red outcomes different from the cue with the highest posterior?

  • Qwen3.5-27B: yes. CoT length does better than the simple rule (0.83 vs. 0.70; difference +0.13 [0.08, 0.18]). Without reasoning, the verbal report (0.66) and opt-out (0.65) also do better (simple rule: 0.52).
  • GPT-OSS-20B: no. At low effort, answer entropy (0.63) and the verbal report (0.61) do about as well as the simple rule (0.64). At high effort, the simple rule does much better (0.85) than the indicators (0.56–0.63). The reason is that GPT at high effort chooses the best-supported cue in 95% of answers, and almost all of the other answers are in histories where these two cues differ.

So when we compare different histories, the indicators add to the count-based rule for Qwen but not for GPT. In Analysis 1, where the history is the same, the result is positive for both models.

AuthorsOpenReview ↗

Author response (2/6)

Weakness 1 and Question 1 (continued): relation to Fleming & Lau [2] and Campbell & Fiske [1]

Sensitivity in Fleming & Lau is formulated as follows: is confidence different when the answer is correct and when it is wrong, with the difficulty of the question held fixed? Analyses 1 and 2 are tests of this kind. In our task "correct" can mean two things, so we report both.

  • Correct = the model chose the cue best supported by the evidence. This asks whether an indicator rises when the model misreads the evidence. Some indicators do. With the same history: CoT length (0.58) and CoT uncertainty words (0.60) for Qwen3.5-27B, and answer entropy (0.67) and the verbal report (0.59) at low effort for GPT-OSS-20B. With similar evidence: 0.65–0.83 for Qwen3.5-27B (0.57 for the verbal report after reasoning) and 0.55–0.63 for GPT-OSS-20B, against 0.52–0.56 for an ideal observer (1 − maximum posterior).
  • Correct = the model chose the true biased cue. In both models, no indicator does better than the ideal observer (differences ≤ 0.04, all CIs include 0). This is expected. The evidence sometimes points to the wrong cue by chance, and the model has no way to know this. This is why we do not use correctness as ground truth.

So some indicators rise when the model itself misreads the evidence. None rises when the evidence happens to point to the wrong cue.

Bias in Fleming & Lau is the overall level of confidence when performance is the same. For the same 300 histories, more reasoning lowers the indicator levels a lot while accuracy stays the same (GPT-OSS-20B answer entropy: 0.73, 0.18, 0.07 for no, low and high effort; accuracy: 0.33, 0.36, 0.35). For this reason we check each setting separately, set a separate threshold for each, and do not compare raw levels across settings or models.

Efficiency in Fleming & Lau is sensitivity relative to task performance (meta-d′/d′). It requires a two-choice task with constant difficulty. Our task has four choices and varying evidence, so we do not estimate it and make no claim about it.

In the terms of Fleming & Lau, some of our indicators show sensitivity in the first sense above, their levels are subject to bias, and we make no efficiency claim.

Campbell & Fiske make two points that apply here: (1) agreement between measures is evidence for a construct only when the measures use different methods, and (2) a construct must also be shown to differ from related ones.

  • Method variance. Our indicators share method variance, and we reported this in Section 5.2: correlations cluster by measurement source (the two CoT indicators; the probe and the SAE score), and we concluded that the indicators are not interchangeable. We do not count agreement within a measurement source as evidence for the construct.
  • Construct variance. The evidence for a shared construct comes from indicators obtained in different ways: a verbal report, an opt-out choice, the answer distribution and the reasoning trace. After removing condition means, all pairs of these output-level indicators remain correlated (r = .23–.81, all significant after Holm correction). This agreement holds across different measurement procedures and cannot come from the evidence conditions.
  • Difference from related constructs. The VERIDICAL–MISLEADING contrast separates the construct from support for the true cue. The identical-input test in analysis 1 above further separates the construct from evidence statistics. As a final note, output-level and representation-level indicators do not correlate once condition means are removed (r ≤ .16), and only the output-level indicators predict the model's next sampling choice. Our claim of a shared construct is therefore limited to the output-level indicators. We describe the probe and SAE scores as readouts that respond to the evidence conditions, and we will revise the text accordingly.

We will add these analyses and this positioning to the paper.

AuthorsOpenReview ↗

Author response (3/6)

Weakness 1: "probe controls"

We agree, and we ran a control for this.

We took the two probes retained in the paper (the GPT-OSS-20B answer-entropy probe and the Qwen3.5-27B uncertainty-language-rate probe). We first predicted each probe target from the evidence statistics alone (M2, the highest and second-highest red proportion, red-minus-green counts, and the number of samples and of red outcomes for each cue). We then subtracted this prediction from the target. What remains is the part of the target that the evidence statistics do not explain, and we trained the probe again to predict this remainder.

The evidence statistics alone predict the targets well (cross-validated r = 0.78 for GPT answer entropy and 0.69 for Qwen uncertainty-language rate). Once their contribution is removed, the probes no longer predict the target: for GPT, r goes from 0.39 to −0.03 (p = 0.71); for Qwen, r goes from 0.37 to 0.06 (p = 0.19). P-values are from permutation tests.

The reviewer is right about the probes. They read the evidence in the history, not uncertainty beyond it. We will report this control, describe the probes in this way, and no longer use them as evidence for an internal uncertainty assessment.

This does not change the answer for the output-level indicators: in Analysis 1 the evidence is identical across the ten repetitions, and some output-level indicators are still higher on the repetitions where the model does not choose the best-supported cue.

Question 2: "equivalence tests"

We assess V–M equivalence using bounds of ±0.3 SD for continuous measures and ±10 percentage points for opt-out rates. The table gives the V − M difference and its 90% CI for every indicator and setting that passes both ambiguity contrasts. Equivalence is established when the whole CI lies inside the bounds. * marks the indicators that entered the composite indices in the submitted version.

ModelIndicatorSettingV − M90% CIEquivalent
GPT-OSS-20BVerbal reportno reasoning+0.02 d[−0.22, +0.25]yes
GPT-OSS-20BVerbal report*low effort−0.22 d[−0.45, +0.02]no
GPT-OSS-20BAnswer entropy*low effort−0.17 d[−0.41, +0.06]no
GPT-OSS-20BAnswer entropyhigh effort−0.02 d[−0.25, +0.22]yes
GPT-OSS-20BCoT lengthlow effort−0.23 d[−0.47, +0.00]no
GPT-OSS-20BCoT length*high effort+0.02 d[−0.22, +0.25]yes
GPT-OSS-20BCoT uncertainty wordslow effort−0.11 d[−0.34, +0.13]no
GPT-OSS-20BCoT uncertainty words*high effort−0.01 d[−0.25, +0.22]yes
GPT-OSS-20BOpt-out*low effort0 pp[−7, +7]yes
GPT-OSS-20BProbe (uncertainty-word density)–+0.07 d[−0.16, +0.31]no
GPT-OSS-20BProbe (uncertainty-word presence)low effort+0.08 d[−0.15, +0.32]no
GPT-OSS-20BProbe (uncertainty-word presence)high effort−0.08 d[−0.31, +0.15]no
GPT-OSS-20BProbe (verbal report)–+0.02 d[−0.21, +0.26]yes
GPT-OSS-20BProbe (answer entropy)*–+0.03 d[−0.21, +0.26]yes
GPT-OSS-20BSAE score*confirmation set+0.01 d[−0.53, +0.54]no
Qwen3.5-27BVerbal reportno reasoning−0.16 d[−0.39, +0.07]no
Qwen3.5-27BVerbal report*reasoning−0.16 d[−0.39, +0.08]no
Qwen3.5-27BAnswer entropyno reasoning−0.43 d[−0.66, −0.20]no
Qwen3.5-27BAnswer entropyreasoning−0.31 d[−0.55, −0.08]no
Qwen3.5-27BCoT length*reasoning−0.26 d[−0.49, −0.03]no
Qwen3.5-27BCoT uncertainty words*reasoning−0.13 d[−0.37, +0.10]no
Qwen3.5-27BOpt-out*reasoning+1 pp[−2, +4]yes
Qwen3.5-27BProbe (uncertainty-word density)–+0.05 d[−0.18, +0.28]yes
Qwen3.5-27BProbe (uncertainty-word rate)*–+0.01 d[−0.22, +0.25]yes
Qwen3.5-27BProbe (answer entropy)–+0.07 d[−0.16, +0.30]no

Equivalence is established for ten indicator–setting combinations: seven for GPT-OSS-20B and three for Qwen3.5-27B. Of the indicators that entered the composites in the submitted version, this holds for four of seven in GPT-OSS-20B and two of five in Qwen3.5-27B. We will limit equivalence claims to the indicators for which equivalence is established. For the others, we will say that a VERIDICAL–MISLEADING difference was not detected, not that the two conditions are equivalent.

Reviewer_4t54OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to measure the internal uncertainty of LLM agents without using answer correctness as ground truth. In a four-cue probabilistic inference task, the authors construct histories in which the ideal observer's posterior is ambiguous, concentrated on the true cue, or concentrated on a wrong cue. They then test whether candidate indicators, ranging from verbal reports and answer distributions to reasoning traces and internal representations, separate these conditions in GPT-OSS-20B and Qwen3.5-27B. The retained indicators converge only partially. Composite indices built from them distinguish the evidence conditions in both models. In Qwen3.5-27B, these indices also decrease as evidence accumulates during model-directed sampling.

Critical strengths

Most work on LLM uncertainty validates its measures against answer correctness, which conflates the model's assessment of the evidence with whether its answer happens to be right. The misleading-evidence condition in this paper separates the two, and I think this is a sound way to approach validation when no ground truth is available.

The paper evaluates behavioural, output-level, reasoning-based and representation-level signals on identical histories under a single set of criteria, which is rarely done. The findings are useful in their own right. The measures are not interchangeable, and their validity depends on the reasoning setting. Some representation-based signals also fail on held-out data. I appreciate that the authors report these negative results openly rather than presenting only the measures that worked.

Critical weaknesses

Every criterion is defined relative to the ideal observer's posterior, which is fully determined by the outcome counts in the history. An indicator that tracks this posterior may only be reading how ambiguous the input looks, not the model's own assessment. Because the model never sees the true cue, any history-based measure would be expected to score veridical and misleading trials similarly, so the discriminant-validity check is weaker than it appears. To support the claim that these measures capture internal uncertainty, the paper should show that they carry information beyond simple features of the history. Alternatively, it could show that they follow the model's own beliefs where these differ from the Bayesian posterior.

Apart from the SAE analysis, indicator screening, variant selection, thresholding and composite evaluation all seem to use the same trials, so the reported composite performance is probably optimistic. The two models were also given different reward instructions, and the GPT-OSS analyses mix prompt versions. This affects the opt-out measure in particular and makes the between-model comparison hard to interpret.

Questions for authors

How well can ambiguous trials be distinguished from concentrated ones using only simple features of the history, such as per-cue outcome counts? How do the retained indicators compare with such a baseline?

A non-significant veridical–misleading difference does not establish equivalence. Could you report equivalence tests for all retained indicators, as you did for the SAE score? The GPT-OSS SAE score appears to have been retained even though equivalence was not established on its confirmation set. Could you explain why?

Some details differ between the main text and the appendix: the sample counts in the evidence-level analysis, the verbal-report threshold for GPT-OSS, and the unit of Qwen's reasoning length. Appendix B also contains a broken section reference. Please reconcile these.

2 comments
AuthorsOpenReview ↗

Author response (1/2)

We thank the reviewer for the positive assessment and for the precise questions, which we address below. They led us to add a baseline built from simple features of the history, two tests of whether the indicators carry information beyond the history, equivalence tests for all indicators, and a re-collection of the GPT-OSS-20B measurements with a single prompt. We report all results, including those that narrow our claims, and will add them to the revised paper.

Weakness 1 and Question 1: "simple features of the history"

We computed several simple features from each trial's history and evaluated how well each distinguishes AMBIGUOUS from VERIDICAL and MISLEADING trials, using average precision (AP; chance = 0.33):

  • Sampling-allocation features (number of cues sampled, maximum samples on a single cue, total red outcomes): AP = 0.24–0.33, i.e. at chance.
  • Highest per-cue red proportion: AP = 0.71.
  • Largest per-cue red-minus-green difference: AP = 0.59.
  • Both heuristics combined (cross-validated logistic regression): AP = 0.75.

The composite indices perform comparably to this baseline for Qwen3.5-27B (AP = 0.73) and better for GPT-OSS-20B (AP = 0.86). Adding either composite to the baseline further improves cross-validated AP (Qwen3.5-27B: 0.83; GPT-OSS-20B: 0.89).

The indicators do correlate with count features, as they should if they follow the evidence. Across the final-round histories of GPT-OSS-20B, the strongest Spearman correlation between a single count feature and an indicator is −.52 (answer entropy with the number of red outcomes on the leading cue). Across the 1,460 trial–rounds of the evidence-level analysis for Qwen3.5-27B it is −.87 (verbal report with the same feature), and at most −.57 for the reasoning-trace indicators.

We agree with the reviewer's point about discriminant validity: because the model never sees the true cue, a similar score in VERIDICAL and MISLEADING is what any history-based measure would give. That contrast shows that the indicators do not follow correctness. It does not show that they carry more than the history. For that we ran two further tests.

  • Do the indicators carry information beyond the history? We held all observable evidence statistics constant. We gave each model the same history ten times. Within each set of repetitions, the input, cue counts, likelihood ratios, and ideal observer posterior are identical, so these quantities cannot explain any variation between the ten responses. We retained histories for which the model did not produce ten identical responses (1,455 of 3,720, or 39%, for GPT-OSS-20B at low reasoning effort; 271, or 7%, for Qwen3.5-27B), and tested whether an uncertainty indicator was higher when the model did not select the cue with the highest ideal observer posterior (AUROC; chance level 0.5). For GPT-OSS-20B at low reasoning effort, answer entropy had an AUROC of 0.67 (95% CI [0.63, 0.71]) and the verbal uncertainty report 0.59 [0.57, 0.62]. For Qwen3.5-27B, reasoning length had an AUROC of 0.58 [0.53, 0.62] and uncertainty-word use 0.60 [0.56, 0.63]. Every confidence interval excludes chance level; for both models, at least some indicators therefore contain information that is not determined by the history. The other indicators tested do not show this pattern (AUROC between 0.20 and 0.53), so we make this claim only for these four.
  • Do the indicators follow the model's own beliefs where these differ from the Bayesian posterior? The test above is about exactly these cases: the model's answer differs from the cue with the highest posterior, and some indicators rise. The model's answers also depart from the posterior in a systematic way. When the cue with the highest share of red outcomes differs from the cue with the highest posterior, GPT-OSS-20B at high effort answers with the former in 14 of 15 such final-round histories, and Qwen3.5-27B in 9 of 15. When we compare histories with similar evidence, Qwen3.5-27B CoT length separates answers that depart from the posterior better than this count rule does (AUROC 0.83 vs. 0.70; difference +0.13 [0.08, 0.18]); the GPT-OSS-20B indicators do not (0.56–0.63 vs. 0.64–0.85).
  • Behaviour. In the model-directed games of Qwen3.5-27B, adding the composite to the count features improves the prediction of whether the model re-checks its preferred cue (AUROC 0.77 → 0.82; difference +0.05 [0.01, 0.08]). For GPT-OSS-20B it does not.

In short: the output-level indicators are not explained by simple count features alone. We will add these analyses to the appendix.

AuthorsOpenReview ↗

Author response (2/2)

Question 2: "equivalence tests"

We now report equivalence tests (TOST; ±0.3 d, and ±10 percentage points for opt-out) for all candidate indicators. Equivalence is established for seven GPT-OSS-20B indicators (verbal report without reasoning; answer entropy, CoT length and CoT uncertainty words with high effort; opt-out with low effort; the verbal-report and answer-entropy probes) and for three Qwen3.5-27B indicators (opt-out with reasoning; the uncertainty-word density and rate probes). For the other indicators that pass both ambiguity contrasts it is not established. Of the indicators in the composites, this covers four of seven in GPT-OSS-20B and two of five in Qwen3.5-27B. We report this next to the original screen, and for the others we now say that a difference was not detected, not that the conditions are equivalent.

Why the GPT-OSS-20B SAE score was retained. The equivalence test on the development set was used only to choose between candidate layers. GPT-OSS-20B had a single candidate layer, so there was no choice to make and the test was not applied. The score was retained by the screening rule used for every indicator (AMBIGUOUS higher than both other conditions, no significant VERIDICAL–MISLEADING difference), which it met on the confirmation set. The V − M difference there is close to zero (+0.01 d), but with about 20 histories per condition the 90% CI ([−0.53, +0.54]) is too wide to establish equivalence. We therefore describe the SAE score as showing no detected VERIDICAL–MISLEADING difference, not as equivalent, and will state this in the revision.

Question 3: "main text and appendix"

Thank you for catching these. All four are corrected in the revision:

  • Sample counts: 146 per bin and 1,460 in total, as in the appendix. Section 4.4 gave the counts before two incomplete observations were excluded.
  • GPT-OSS-20B verbal-report threshold: ≥ 12.5, as in Table 1. Equation 14 was left over from an earlier analysis and is removed.
  • Qwen3.5-27B reasoning length: measured in tokens, as in Section 4.1.3 (threshold ≥ 2,198). Appendix D.4 and Table 1 used an earlier character-based version; the two measures correlate at r = .995.
  • Appendix B: the broken reference now points to Section 3.3.

Weakness 2: "reward instructions and prompt versions"

We agree; this was an unintended prompt-version mismatch. We have re-collected the affected GPT-OSS-20B measurements (verbal reports, probe activations, Jacobian-lens readout) with the prompt used for Qwen3.5-27B, and the conclusions do not change: the verbal-report effects are 0.80 and 0.85 without reasoning (previously 0.81 and 0.87), and the probe correlation is r = 0.39 with both prompts.

Weakness 2: "same trials for selection and evaluation"

We agree that the reported composite performance is relatively optimistic, and we will say so. Two results do not depend on this: the indicator counts use cross-fitted thresholds (five folds, repeated 20 times), and the model-directed games are separate data, with thresholds fitted on the synthetic histories and applied to these games unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.