- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors point out that existing research has largely focused on the accuracy and calibration of the output probabilities generated by language models; however, even when these probability predictions are accurate, it does not necessarily mean they can explain the model’s actions. To address this, the paper proposes a black-box evaluation framework based on decision theory. Without requiring a pre-specified utility function, the framework examines whether the elicited beliefs are consistent with the model’s decision-making behavior.
Critical strengths
-
The research question is meaningful and clearly distinguished from traditional probability calibration.
-
It does not require prior knowledge of specific utility coefficients, which represents a valuable theoretical design.
-
It provides a clear theoretical distinction between necessary conditions and sufficient conditions at the observational level.
-
The appendix includes a number of robustness checks.
Critical weaknesses
1.The framework relies on a restrictive utility assumption, despite claiming not to assume the agent's utility function.
The paper repeatedly emphasizes that it can verify elicited beliefs without specifying a utility function. However, this does not mean it makes no assumptions regarding the decision-maker's utility. A univariate hierarchical approach cannot rule out patient-specific losses determined by the joint interaction of multiple variables; furthermore, specifying costs within a prompt does not necessarily guarantee that the model has actually adopted that exact same utility structure.
2.The main test fails to adequately distinguish between noisy probability elicitation and meaningful belief–action inconsistency. Although the authors experiment with alternative prompts, calibration methods, and various statistical estimators, these approaches do not directly address the issue of measurement error.
I would like to see two control analyses: (1) Recalculate the CI (Consistency Index) and CMI (Consistency-Mismatch Index) using the mean of multiple probability reports for the same x, and observe how the conclusions change. (2) Use a synthetic agent—one with a known latent belief that guides its actions—and introduce noise or rounding (similar to that observed in LLMs) into its probability reports; then, examine the extent of the bias produced by the authors' test. The core issue is not whether the main test statistically rejects the null hypothesis, but rather to what degree the magnitude of this rejection actually reflects a genuine belief–action mismatch.
3.The cyclic-monotonicity test is too permissive to establish meaningful action-guiding behavior
How would the test perform if it employed random actions, actions that remain nearly invariant to risk, or decision rules that—while running counter to medical logic—nonetheless possess a coherent mathematical utility ordering? In Appendix H.2, the authors evaluate the statistical power of the test against certain synthetic patterns of violation; however, they do not sufficiently address "negative controls"—that is, scenarios involving weak reliance on beliefs or behaviors that are logically unrelated to the medical context.
4.Observational rationalizability does not establish that elicited beliefs explain the model's actual decision process
If two agents exhibit identical performance within aggregated probability–outcome–action data, current theory cannot necessarily distinguish between them, even if they rely on different patient characteristics to determine their actions. A simple counterexample is a scenario where an agent’s actions are influenced by a contextual feature unrelated to the disease state, while the reported probabilities reflect only the risk of the disease. Such an action mechanism might pass tests based on aggregated statistics, yet this does not imply that the reported probabilities provide a complete account of the actual action policy.
Questions for authors
same as weaknesses
3 comments
Thanks for the responses. I would like to raise my score.
Thanks
Thanks so much for your response, please let us know if you have any other questions or concerns that we can address!
Rebuttal
Thank you for the thoughtful critique. We ran every control you proposed; all are in the new Appendix I, and the framework behaves as intended in each of them.
W1: The utility assumption. Our only restriction is that utility depends on the action and outcome, not directly on the evidence; u(a,θ) itself is unrestricted. This restriction is unavoidable: if u could depend freely on x, any behavior would be rationalizable without beliefs (§3.1). On joint interactions, the calibrated test now also adjusts for every covariate, and every pair of covariates, revealed in at least 15 cases. For the six flagged pairs, the violation remains in 81 of 84 single-covariate and 610 of 642 two-covariate adjustments (I.4). A patient-specific loss that explained it would have to depend on three or more covariates jointly and also track the true risk better than the model's own report. Models do respond to stated costs: raising the stated deferral cost from 0.5 to 2.0 cuts GPT-High's deferral from 0.31 to 0.07. With costs stated, the calibrated test still rejects in 9 of 15 pairs (H.1).
W2: Noise versus genuine inconsistency. (1) Averaged reports. The calibrated test now conditions on the mean of each case's five reports, which is your proposed control, and treats cases as the units (§3.2). Violations remain for Llama in all four domains and for GPT-5 on Heart (p≤0.014). (2) Synthetic agents with a known latent belief (I.2). With each pair's observed noise and rounding, single-report tests reject for 46% of noise-only datasets, while our test rejects for 9.9% (at most 7.7% for five of the six flagged pairs). Noise-only gaps shrink to 0.58 when reports are averaged; the flagged gaps do not (ratio 0.89–1.31). The rejections therefore reflect belief–action mismatch. Where noise does contribute, as for DeepSeek-R1 on Heart, whose gap halves with averaging, the test does not flag the pair.
W3: Negative controls (I.3, Table 14; 800 datasets per agent).
W4: Context-driven actions. We agree the tests establish observational equivalence, as §3 states, and we added your counterexample (I.3). If actions depend only on a disease-unrelated feature, the margin is never positive, so the report is flagged as not action-guiding. If actions depend on the report and the feature, the feature carries no information about the disease beyond the report. It then acts like a random-utility shock allowed by Definition 1, and passing sufficiency (5.8%) is correct. To detect such agents anyway, Definition 1 implies A⊥X∣PS, which we test with the feature in place of the outcome. This check needs no labels and detects both agents in 100% of datasets.