acceptodds
Under review as a conference paper at ICLR 2027

When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs

Abstract

Large language models (LLMs) are increasingly deployed in high-stakes settings where good decisions require forming beliefs over the probability of unknown outcomes. However, it is unclear whether LLMs act as if they hold coherent beliefs when making decisions, or if so, how we could validate models' reports of such beliefs. We propose a decision-theoretic framework that elicits both probability judgments and decisions from an agent and tests their mutual consistency. Formally, our methods characterize whether it is possible for the actions to be produced by a “near-rational" decision maker who holds the elicited probability as their true belief. We show that, perhaps surprisingly, this formalization implies empirically testable conditions even without any assumption about the agent's utility function. Applying our framework to stylized clinical diagnosis tasks, we find that models' reported beliefs are demonstrably imperfect summaries of the information revealed in their decisions, but that the discrepancies are small for the strongest models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Decision Making, Beliefs, Preferences

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TUtuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors point out that existing research has largely focused on the accuracy and calibration of the output probabilities generated by language models; however, even when these probability predictions are accurate, it does not necessarily mean they can explain the model’s actions. To address this, the paper proposes a black-box evaluation framework based on decision theory. Without requiring a pre-specified utility function, the framework examines whether the elicited beliefs are consistent with the model’s decision-making behavior.

Critical strengths
  1. The research question is meaningful and clearly distinguished from traditional probability calibration.

  2. It does not require prior knowledge of specific utility coefficients, which represents a valuable theoretical design.

  3. It provides a clear theoretical distinction between necessary conditions and sufficient conditions at the observational level.

  4. The appendix includes a number of robustness checks.

Critical weaknesses

1.The framework relies on a restrictive utility assumption, despite claiming not to assume the agent's utility function.

The paper repeatedly emphasizes that it can verify elicited beliefs without specifying a utility function. However, this does not mean it makes no assumptions regarding the decision-maker's utility. A univariate hierarchical approach cannot rule out patient-specific losses determined by the joint interaction of multiple variables; furthermore, specifying costs within a prompt does not necessarily guarantee that the model has actually adopted that exact same utility structure.


2.The main test fails to adequately distinguish between noisy probability elicitation and meaningful belief–action inconsistency. Although the authors experiment with alternative prompts, calibration methods, and various statistical estimators, these approaches do not directly address the issue of measurement error.

I would like to see two control analyses: (1) Recalculate the CI (Consistency Index) and CMI (Consistency-Mismatch Index) using the mean of multiple probability reports for the same x, and observe how the conclusions change. (2) Use a synthetic agent—one with a known latent belief that guides its actions—and introduce noise or rounding (similar to that observed in LLMs) into its probability reports; then, examine the extent of the bias produced by the authors' test. The core issue is not whether the main test statistically rejects the null hypothesis, but rather to what degree the magnitude of this rejection actually reflects a genuine belief–action mismatch.


3.The cyclic-monotonicity test is too permissive to establish meaningful action-guiding behavior

How would the test perform if it employed random actions, actions that remain nearly invariant to risk, or decision rules that—while running counter to medical logic—nonetheless possess a coherent mathematical utility ordering? In Appendix H.2, the authors evaluate the statistical power of the test against certain synthetic patterns of violation; however, they do not sufficiently address "negative controls"—that is, scenarios involving weak reliance on beliefs or behaviors that are logically unrelated to the medical context.


4.Observational rationalizability does not establish that elicited beliefs explain the model's actual decision process

If two agents exhibit identical performance within aggregated probability–outcome–action data, current theory cannot necessarily distinguish between them, even if they rely on different patient characteristics to determine their actions. A simple counterexample is a scenario where an agent’s actions are influenced by a contextual feature unrelated to the disease state, while the reported probabilities reflect only the risk of the disease. Such an action mechanism might pass tests based on aggregated statistics, yet this does not imply that the reported probabilities provide a complete account of the actual action policy.

Questions for authors

same as weaknesses

3 comments

Thank you for the thoughtful critique. We ran every control you proposed; all are in the new Appendix I, and the framework behaves as intended in each of them.

W1: The utility assumption. Our only restriction is that utility depends on the action and outcome, not directly on the evidence; u(a,θ)u(a,\theta) itself is unrestricted. This restriction is unavoidable: if uu could depend freely on xx, any behavior would be rationalizable without beliefs (§3.1). On joint interactions, the calibrated test now also adjusts for every covariate, and every pair of covariates, revealed in at least 15 cases. For the six flagged pairs, the violation remains in 81 of 84 single-covariate and 610 of 642 two-covariate adjustments (I.4). A patient-specific loss that explained it would have to depend on three or more covariates jointly and also track the true risk better than the model's own report. Models do respond to stated costs: raising the stated deferral cost from 0.5 to 2.0 cuts GPT-High's deferral from 0.31 to 0.07. With costs stated, the calibrated test still rejects in 9 of 15 pairs (H.1).

W2: Noise versus genuine inconsistency. (1) Averaged reports. The calibrated test now conditions on the mean of each case's five reports, which is your proposed control, and treats cases as the units (§3.2). Violations remain for Llama in all four domains and for GPT-5 on Heart (p≤0.014p\le0.014). (2) Synthetic agents with a known latent belief (I.2). With each pair's observed noise and rounding, single-report tests reject for 46% of noise-only datasets, while our test rejects for 9.9% (at most 7.7% for five of the six flagged pairs). Noise-only gaps shrink to 0.58 when reports are averaged; the flagged gaps do not (ratio 0.89–1.31). The rejections therefore reflect belief–action mismatch. Where noise does contribute, as for DeepSeek-R1 on Heart, whose gap halves with averaging, the test does not flag the pair.

W3: Negative controls (I.3, Table 14; 800 datasets per agent).

  • Random and risk-invariant agents. These pass sufficiency at the nominal rate (5.2% and 0%), correctly, since their actions carry no information about the outcome. But their monotonicity margin is never significantly positive (0%), so they fail the second requirement, that beliefs guide action.
  • A coherent rule against medical logic. It passes both tests, as it should: the framework validates beliefs under any preferences. Its fitted utility differences rank No above Yes in the diseased state in 100% of datasets, which exposes the preference. All 13 real pairs with a positive margin rank Yes above No.
  • Power. Acting against the report in two bands is detected in 99.9% of datasets, and an agent that knows more than it reports is caught in 99.4%.

W4: Context-driven actions. We agree the tests establish observational equivalence, as §3 states, and we added your counterexample (I.3). If actions depend only on a disease-unrelated feature, the margin is never positive, so the report is flagged as not action-guiding. If actions depend on the report and the feature, the feature carries no information about the disease beyond the report. It then acts like a random-utility shock allowed by Definition 1, and passing sufficiency (5.8%) is correct. To detect such agents anyway, Definition 1 implies A⊥X∣PSA\perp X\mid P_S, which we test with the feature in place of the outcome. This check needs no labels and detects both agents in 100% of datasets.

Thanks for the responses. I would like to raise my score.

Thanks so much for your response, please let us know if you have any other questions or concerns that we can address!

Reviewer_kD1yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks an interesting question and offers a useful framework for separating calibration from behavioral consistency. My remaining concerns are the precise scope of the characterization and the interpretation of the smaller empirical violations. The robustness checks in Appendix H strengthen the evidence, but do not fully resolve these points. I currently lean toward rejection, and would reconsider if the authors clarify the theoretical assumptions and show that the main findings survive appropriate null calibration and elicitation-noise checks.

Critical strengths
  • The distinction between probability calibration and behavioral consistency is useful. An accurate probability report need not explain how a model chooses its actions.

  • Appendix H examines stated costs, alternative prompts, different estimators, and context-level resampling. These checks help identify which empirical findings persist across analysis choices.

Critical weaknesses
  • Proposition 5 explicitly relaxes the strict convexity required by Definition 1, so it does not establish sufficiency for the original model class. In the relaxed class, the necessity argument also requires a belief-only selection rule when multiple maximizers exist. Membership in the argmax alone does not ensure conditional independence.

  • The statistical evidence needs stronger calibration. A positive bootstrap interval for estimated CMI does not by itself establish a valid conditional-independence test. Appendix H adds useful checks, but permutation within probability deciles is not an exact test conditional on the continuous probability. Separately elicited, noisy or rounded reports may also leave residual dependence under a coherent underlying policy.

Questions for authors
  • Does the sufficiency result extend to the strictly convex class in Definition 1? If not, what additional conditions are required, and how is selection among tied maximizers restricted in the relaxed class?

  • What false-positive rates do the CMI tests achieve on simulated agents satisfying the stated null, with matched sample sizes and context-level repetitions? Appendix H reports power against violations, but this does not establish control of false positives.

  • Since independence conditional on a continuous probability does not imply independence within probability deciles, how sensitive are the permutation results to bin width and residual variation within bins?

  • How much residual action-outcome dependence can probability rounding and repeated-elicitation noise produce under an otherwise coherent policy? Can these effects account for the smaller violations reported for the strongest models?

5 comments

Thank you for a precise and constructive review. We addressed both conditions you set: the theory is clarified, and the sufficiency results now rest on a test calibrated on simulated nulls and checked against elicitation noise.

W1/Q1: Strict convexity and tie-breaking. We stated Proposition 5 for convex rather than strictly convex CC deliberately, so that it covers exact expected-utility maximization (C≡0C\equiv0) as well as approximate maximization. This does not change how the tests are interpreted: either agent holds the elicited belief and acts on it. You are right that this wider class needs a tie-breaking condition: argmax membership alone does not give conditional independence, since an agent indifferent between actions could use xx to choose among them.

  • Expanded Definition 1. We therefore expand Definition 1 to this wider class and build the condition into it: CC need only be convex, and choice probabilities depend on xx only through PSP_S, which is automatic when CC is strictly convex. All of our results apply to this class (the strict part of Proposition 3 to strictly convex CC), and Proposition 5 characterizes it exactly; its proof already used this property.
  • Strictly convex CC. Where it is useful to exclude exact utility maximization, the new Corollary 1 shows that if, in addition, the monotonicity of Proposition 3 holds strictly (which γ^>0\hat\gamma>0 tests, Proposition 4) and choice probabilities vary continuously with the belief, the regularizer can be taken strictly convex. Continuity is what separates approximate from exact maximization: a strictly convex CC cannot make choices jump, whereas an exact maximizer switches actions abruptly where it is indifferent. With finite-precision belief reports (as would be outputted by an LLM), continuity holds automatically, since the agent's choices at unobserved beliefs between observed values can be filled in continuously.

W2/Q2: Null calibration. We agree that an interval excluding zero is not by itself a conditional-independence test, so we now base inference on a test calibrated on simulated nulls. Our null agents keep each pair's cases, five queries per case, reports, reference risks, and action rates, and replace only the actions (see I.1, Table 11). Permutation criteria that condition on a single report, such as permutation within deciles, hold 4–5% when queries are independent, but reject 55–60% of null datasets once actions vary by case for reasons unrelated to the outcome. The interval criterion rejects about 9% of null datasets on average, more for a few pairs. We therefore test with cases as the units. For each case we average its reports, residualize its reference risk and action shares on a cross-fitted spline of this mean (a generalized covariance measure; Shah & Peters, 2020), and permute action shares among cases with similar mean reports. Across three null designs, this test rejects 4.6–5.5% (no template above 8%). Under it, sufficiency is violated for Llama in all four domains and for GPT-5 at both reasoning levels on Heart (p≤0.014p\le0.014). DeepSeek-R1 shows no violation in any domain.

Q3: Bin width. Under independent queries, permutation within 5, 10, or 20 belief bins holds 4.6%, and permutation among identical reports 4.4%. When each case reports a stable belief, coarse bins are slightly liberal (5.7–6.1%, against 4.6% for identical reports). The case-level test does not assume that the belief is constant within a bin: the spline removes smooth trends, and permutation handles the rest. On the real data, every setting with 10, 20, or 30 bins and 16, 24, or 32 knots flags the same six pairs, and the 30-bin setting flags one more (Table 12). A less flexible 8-knot spline over-rejects for one pair whose actions switch sharply with its belief, so we use 24 knots.

Q4: Rounding and repeated-elicitation noise. We simulated coherent agents whose reports equal a latent belief plus each pair's observed within-case noise (SD 0.04–0.18), rounded to its grid, with actions that depend only on the latent belief (I.2). A test on single reports rejects for 46% of these noise-only datasets, so your concern is warranted for single reports. Our case-level test averages five reports and rejects for 9.9% (10.5% with rounding to 0.1). For five of the six flagged pairs, noise alone triggers it at most 7.7% of the time. A check on the real data points the same way. Averaging mm reports cuts noise variance by 1/m1/m, so a gap caused by noise must shrink as mm grows. Noise-only gaps shrink to 0.58 of their single-report value, while the flagged gaps do not shrink (ratio 0.89–1.31). For the strongest model, DeepSeek-R1, the single-report gap on Heart halves with averaging (0.44 to 0.22). So for the strongest model, noise can explain the smaller violations, consistent with the paper's conclusion that the strongest models are close to coherent.

Hi, just wanted to send a reminder that we posted our rebuttal. Looking forward to your feedback and thanks again for taking the time to review!

Hi! Just following up again to see if we addressed your concerns. We remain thankful of the work you have put in thus far into reviewing our paper.

Reviewer_1YWbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a decision-theoretic framework that prompts agents to provide probability judgments and decisions from an agent and tests their mutual consistency. Tests are parformed on a binary medical diagnosis tasks with GPT-5, DeepSeek R1, and Llama 4. The study concludes that stated probabilities are informative but imperfect summaries of decision-relevant information.

Critical strengths

Important problem, analysed through a well defined framework. The framework require no assumptions about the model’s utilities, only black-box access, so it can be applied to closed models.

Thorough experimental setup. Results are checked across estimators, permutation nulls, stated payoffs, stratification and alternative prompts. The authors are explicit that passing the tests means failing to falsify the belief report, not proving the model holds the belief.

Critical weaknesses
  1. The practical value and generality of the test is not clear to me. It is my understanding that the test requires ground-truth outcomes, the same data one would use to evaluate accuracy and calibration directly. 2 It seems that results do not carry over across domains. For example, the same model passes in one domain and fails in another (e.g., DeepSeek: 22.4% RF improvement on Heart vs. −3.5% on Cry). The authors do recommend “task- and model-specific validation”. This warrants even more serious questions, e.g., whether within the same domain results can hold, in cases with a domain drift in the data (e.g., 2 different hospitals, or new population in a hospital, or new disease)
Questions for authors
  1. How general is the framework beyond the evaluated setting? The monotonicity test is implemented only for binary outcomes, and inputs are short structured descriptions. How would it apply to multi-class outcomes or free-text inputs (e.g., clinical notes), and how many cases and how much variation in actions are needed for the tests to have power?
  2. Do results hold within a domain under data drift? The heart-disease data include care setting (inpatient, outpatient, emergency, procedural) and demographic strata. Could you establish validity on one subpopulation and test it on another, as a proxy for a new hospital or patient population? Does validity established for one condition carry over to a new condition for the same model?
  3. Given the recommendation for “task- and model-specific validation,” would validity have to be re-established with labeled outcomes for every new domain, population or condition? If so, how do you envision the framework being used in practice, and how much labeled data would each re-validation require?
5 comments

Thank you for these questions, which the new Appendix I answers. We hope our new analysis and answers provide the necessary clarification.

W1: Ground truth and differences across domains. Differences across domains are what a validation tool should reveal: validity is a property of a model on a task, like calibration or accuracy. The calibrated test makes the pattern sharp. Llama violates sufficiency in all four domains, GPT-5 only on Heart, and DeepSeek-R1 in none. For three of the four models, the size of the violation differs significantly across domains (p≤0.018p\le0.018; I.5). Labels are needed only for the sufficiency test, and Q3 gives the cost.

Q1: Generality and power. Propositions 1, 2, and 5 hold for any finite sets of states and actions; Propositions 3 and 4 and Corollary 1 are stated for two states (I.7). With more states the belief is a vector, and the sufficiency test residualizes on a smooth function of it. Cyclic monotonicity with an unknown utility matrix remains a linear program: by Rockafellar's theorem it holds exactly when there are potentials with ϕn≥ϕm+qm⋅U(bn−bm)\phi_n\ge\phi_m+q_m\cdot U(b_n-b_m), which is linear in (U,ϕ)(U,\phi). Free-text inputs such as clinical notes enter only through the prompt; nothing in the tests depends on the form of the context. On power, the six observed violations reach 80% power with about 100–400 cases of five queries each (Table 17). Action variation matters as much as case count: a model that defers on 98.6% of cases reveals little at any sample size.

Q2: Drift within and across conditions. We split each real-patient domain by a revealed covariate (sex, age, care setting, and ethnicity for Heart; sex, mobility, income, and self-rated health for Diabetes) and estimated the violation in each half (I.5, Table 16). Its direction transfers: the gap has the same sign in both halves in 22 of 23 splits. Its size can change (7 of 23 differ at the 5% level, uncorrected). Across conditions, transfer cannot be assumed; for instance, GPT-5 violates sufficiency on Heart but not on Fever or Diabetes. Validity established on one condition should therefore be re-checked on a new one.

Q3: Labels per re-validation and use in practice. Only the sufficiency test needs labels. Monotonicity and the fitted utility ordering use reports and actions alone. To detect violations of the sizes we observe, the sufficiency test needs about 100–400 cases labelled with reference risks, or several hundred to over a thousand with one binary outcome per case (I.6). In practice: (i) check monotonicity and the fitted ordering on unlabelled queries; (ii) label a few hundred cases and run the calibrated test; (iii) repeat when the condition or population shifts (I.6).

Hi, just wanted to send a reminder that we posted our rebuttal. Looking forward to your feedback and thanks again for taking the time to review!

I thank the authors for concise response and the new analyses in Appendix I. The calibrated case-level test, the noise controls and the negative controls strengthen the method, and the drift analysis (I.5) answers my Q2 directly.

The answers also confirm my main concern about practical scope. Validity does not transfer across conditions, the size of the violation changes across subpopulations in several splits, and each new condition or population requires re-validation with several hundred labeled cases. Multi-class outcomes and free-text inputs are argued theoretically but not demonstrated. I agree that this applies to any validation method, including calibration, but it should be stated clearly in Section 6, so that the test is presented as a complement to task-specific validation rather than a replacement for it.

Overall, my concerns have been answered, and I confirm my rating of 3 (weak acceptance).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments

Thank you for the careful and constructive reviews. We have uploaded a revised PDF, with all changes in blue. The new analyses are in Appendix I and the new theory in Corollary 1.

What changed

  • Theory. As Proposition 5 deliberately allowed for exact utility maximization, we have made a simple expansion to the class of choice models in Definition 1 to cover exact as well as approximate expected-utility maximization, with the tie-breaking condition this wider class needs. All our results apply to it, and the interpretation of the tests is unchanged. A new Corollary 1 adds that when monotonicity also holds strictly, which our margin γ^>0\hat\gamma>0 tests, the regularizer can be taken strictly convex.
  • A calibrated sufficiency test (I.1). We simulated agents that satisfy the null while matching each dataset's cases, repeated queries, reports, and action rates. Criteria that treat the five queries of a case as independent over-reject once actions vary by case for reasons unrelated to the outcome. We therefore now test with cases as the units: we average each case's reports and permute cases with similar mean reports. This test rejects 4.6–5.5% of null datasets.
  • Findings under this test. Sufficiency is violated for Llama in all four domains and for GPT-5 at both reasoning levels on Heart (p≤0.014p\le0.014). In these pairs, at the same mean report, cases the model always calls positive carry 13–68 points more reference risk than cases it never does. DeepSeek-R1 shows no violation in any domain. These are largely the pairs where the paper's Random Forest comparison found the largest gains, and the pattern sharpens the paper's conclusion that the strongest models are closest to coherent.
  • Noise and rounding (I.2). The flagged violations do not shrink as more reports are averaged, whereas those of simulated noise-only agents do.
  • Further checks. Negative controls and context-driven agents (I.3), adjustment for one or two covariates jointly (I.4), transfer across subpopulations and conditions (I.5), labelled-data requirements (I.6), and scope beyond binary outcomes and structured inputs (I.7).
  • Main text. The calibrated test (§3.2, Table 1, §5), a summary of the new checks (§5), and practical guidance and scope (§6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.