- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for diagnosing probabilistic reasoning in LLMs by decomposing total decision regret into belief loss and a residual. The framework distinguishes errors in estimating probabilities from errors in translating reported beliefs into actions under specified loss functions. The authors evaluate this decomposition across several LLM families and reasoning settings using synthetic tasks. They further investigate four RL interventions targeting belief formation, decision-making, or both. Experiments across synthetic inference, weather forecasting, and clinical reasoning show that improving beliefs does not necessarily improve decisions, while decision-only training can improve actions without substantially improving reported beliefs. Joint training improves both components, although its effectiveness depends on the training and evaluation formats.
Critical strengths
-
The paper provides a clear decision-theoretic framework for distinguishing errors in belief formation from errors in translating beliefs into actions. The regret decomposition is simple and interpretable, and the use of controlled decision problems with known posteriors allows the authors to systematically investigate different sources of decision failures. This provides a useful perspective beyond evaluating confidence calibration or final decision accuracy alone.
-
The paper systematically evaluates four RL interventions across three domains and two elicitation formats. An interesting finding is that improving belief accuracy does not necessarily lead to better decisions, while decision-only training can improve decision performance without comparable improvements in reported beliefs. The comparison between independent and two-turn evaluation further highlights the importance of belief-action alignment and training-evaluation format consistency. These findings offer useful insights into designing post-training objectives for probabilistic reasoning.
Critical weaknesses
-
The decomposition relies on the model's verbalized belief to distinguish belief formation errors from decision optimization errors. However, reported beliefs may not faithfully represent the model's internal beliefs. Consequently, the optimization residual may reflect inconsistencies between elicited and internally used beliefs rather than an actual failure to optimize decisions under a given utility function. Moreover, the optimization residual is defined as a signed quantity and can be negative when the reported action performs better than the action implied by the reported belief. Therefore, improvements in the two components do not necessarily correspond to independent improvements in belief formation and decision optimization. This complicates the interpretation of the proposed diagnostic framework, particularly when attributing performance changes to specific reasoning capabilities.
-
The post-training experiments are conducted on a single base model (Qwen3-8B), and the decision problems are restricted to binary outcomes with relatively simple threshold-based loss functions. Although the paper evaluates three domains and includes generalization to unseen thresholds, the training and evaluation tasks largely share the same underlying structures. In particular, decision-only training may learn task-specific decision rules rather than generalizable probabilistic reasoning abilities. Similarly, the strong dependence of sequential alignment training on matching elicitation formats raises concerns about whether the improvements reflect robust belief-action alignment or adaptation to a specific prompting procedure.
Questions for authors
-
How can the authors distinguish genuine decision optimization errors from discrepancies between verbalized beliefs and the beliefs internally used by the model? Since the optimization residual can be negative, to what extent can it be interpreted as a separate source of decision-making error rather than a measure of belief-action inconsistency?
-
The sequential alignment intervention improves performance primarily when the evaluation format matches the training procedure. Can the authors clarify whether this improvement reflects a generalizable belief-action alignment capability or simply the model learning to condition its decisions on explicitly reported probabilities?
-
How well do the RL interventions generalize to decision problems with different structures beyond the threshold-based binary classification setting? While the experiments consider unseen thresholds, these share the same decision structure as the training tasks.
-
Are the observed effects consistent across different base models? The diagnostic experiments include multiple model families, but all post-training experiments use Qwen3-8B. Evaluating at least one additional model would help establish whether the reported patterns are specific to the chosen architecture or initialization.
7 comments
Q1 - How the framework reflects genuine optimization errors
To directly answer your question on whether measured optimization residuals mainly reflect genuine optimization errors or belief-action misalignment, we have added a supplementary experiment.
We evaluate the decision consequences when a model is directly provided with the beliefs, and must only choose the best action. We compare three conditions: 1) independent evaluation, which elicits beliefs and actions in separate sessions; 2) two-turn evaluation, which gives the model access to evidence and its reported beliefs; and 3) belief-only evaluation, which directly supplies the reported beliefs to the model and asks for actions without the original evidence. We evaluate mean optimization residuals and agreement between belief-implied and reported actions. We test GPT-5.5 in both thinking and non-thinking modes and provide the results in the following table.
We find that with medium thinking effort, the model reliably chooses belief-implied optimal actions when probabilities are explicitly available. This suggests that in this setting, optimization residuals in independent evaluation may mainly reflect belief-action misalignment rather than a general inability to optimize under supplied probabilities. On the other hand, we find that without thinking, belief–action agreement remains between 87.0% and 89.9% across the three conditions. This may reflect genuine difficulty optimizing the given utility functions, although we cannot rule out the possibility that the model does not fully adopt the supplied beliefs.
In revision, we will explicitly clarify that our framework can be supplemented with further comparisons to tease apart more specific contributors to belief loss or optimization residuals. We will also add these results to Section 4, along with evaluations on more models.
Table 3. Changes in decision consequences when explicitly providing beliefs
| Thinking mode | Evaluation format | Mean | Action agreement |
|---|---|---|---|
| None | Independent | 0.018 | 87.0% |
| None | Two-turn | 0.023 | 89.9% |
| None | Belief-only | 0.022 | 87.7% |
| Medium | Independent | 0.0043 | 98.4% |
| Medium | Two-turn | 0.0003 | 99.8% |
| Medium | Belief-only | 0.0003 | 100% |
W1-2 - Distinguishing independent improvements
We appreciate the reviewer raising the question on how much of the improvement in regret-based terms can be interpreted as independent improvement in separate capabilities.
It is correct that the two errors will trade off as a consequence of using the reported beliefs as the reference. We use these regret terms as our primary evaluation metrics because they quantify how interventions affect different sources of decision loss against ground truth optimal actions, which is arguably the biggest concern when relying on LLMs for decision-making.
Assessing improvements to other metrics—such as MSE for belief accuracy and agreement between belief-implied and reported actions for belief–action alignment —can certainly help corroborate our claims about the effectiveness of post-training. We have added both analyses. Overall, these secondary metrics show changes consistent with the regret-based findings, providing additional support for the observed intervention effects.
It should be noted, however, that improvements in these direct metrics will not translate into reductions in decision loss. For example, improving a belief estimate from 0.4 to 5, given the true posterior is 0.7, would improve belief accuracy but would not change the belief-implied action at a decision threshold of 0.6. The decision loss associated with that action would therefore remain unchanged, which is why we center our framework on decision consequences.
We will update the results with supplementary metrics in the revision. And we will also discuss the trade-offs between different metrics further with the added space in the revision, showing how additional metrics can be brought in to complement the information provided by our framework.
W1-1 Discrepancy between verbalized and internal beliefs and how to interpret the framework
Thank you for raising the important point that discrepancies between verbalized and internal beliefs can complicate distinguishing genuine decision optimization errors from belief-action inconsistency.
Discrepancy between verbalized and internal beliefs
We wish to note that a model’s true belief is latent, thus we cannot rule out measurement errors. But the value of our framework does not depend on the reported belief being the true belief: rather, a goal of our work is to show how threats to LLMs’ decision-making can be evaluated despite a lack of access to a model’s true beliefs.
Ultimately, there is no way to separate errors in beliefs from errors in optimization without some intermediary quantity that can be measured. Our framework uses reported beliefs as this intermediary quantity, because they are easily accessible in practice and apply to both open-source and proprietary models. We evaluates whether that choice is useful by showing how the quantities in the framework can be used to drive interventions or identify patterns across models.
We currently acknowledge the potential difference between reported and true beliefs in lines 116-118 of the paper. In revision, we will further clarify why our approach does not depend on accessing true beliefs for the purpose of providing meaningful measurement of failure modes and targets for post-training.
Interpretation of belief loss and optimization residuals
We would also like to use this opportunity to clarify the interpretation of belief loss and optimization residuals.
Belief loss refers to the decision loss attributable to the reported beliefs differing from the true posterior beliefs. It summarizes several potential sources of loss before the model reports its belief, including errors in extracting information from evidence, unexpected ways of using its prior knowledge and elicitation distortion.
Similarly, optimization residual captures the decision consequences of actions being inconsistent with the optimal implied by reported beliefs. It indicates all possible changes from belief-implied to reported actions, including failures to optimize the given utility function and it using beliefs different from those they report. That is why we decided to name the term as optimization residual rather than loss.
In revising, we will add additional emphasis on the fact that these terms represent broad distinctions between error sources, which additional comparisons (as we discuss below) can extend to further disambiguate sources of loss.