acceptodds
Withdrawn from ICLR 2027

Training Language Models to Be Coherent Decision-Makers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable decision-making requires more than accurate prediction: a model must preserve its beliefs, apply the relevant utilities, and recognize when the information needed to justify an action is missing. We study whether language models can learn this decision procedure from supervised fine-tuning and generalize it across domains and differing natural-language expressions of the decision challenge. Across 20 datasets, we explore challenges of belief instability and decision-making errors by first eliciting probabilities of outcomes and then varying only the utilities and the framing of the decision problems, while holding the evidence fixed. We train models to preserve elicited beliefs while selecting the action that maximizes expected utility, and evaluate transfer to unseen application domains, held-out framings, and different classes of payoff structures. We further introduce incomplete-information settings in which required utilities are withheld and replaced with irrelevant text, testing whether models can distinguish missing decision-relevant information from merely additional context. We find that targeted fine-tuning substantially improves coherent decision-making and that in many situations, learning transfers across domains and framings to situations unobserved during training. Further, models trained for decidability learn to identify when action cannot be justified based on missing information. Finally, we show the value of a routed system that considers separately the recognition of decision completeness and utility-sensitive decision execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Decision-Making, Trustworthy Reasoning, Beliefs, Utilities

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_7BGNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether small language models can be fine-tuned to behave as coherent decision-makers, where coherent decision-making requires both preserving beliefs under changes in utilities and correctly mapping beliefs and consequences to actions. The authors construct a benchmark based on 20 tabular binary-prediction domains and systematically vary decision utilities and natural-language framings while keeping the underlying evidence fixed. They evaluate belief distortion and belief-conditional decision errors separately, and further investigate incomplete-information settings in which one or both error costs are withheld. Across six Gemma and Qwen models, the authors find that targeted supervised fine-tuning substantially improves decision coherence and belief stability, with some transfer to unseen domains and framings. They also find that models specialized for recognizing missing decision-relevant information perform better at decidability, while complete-information CoT specialists are better at executing decisions, motivating a routed system that combines the two capabilities. Overall, the paper presents a carefully controlled framework for disentangling belief instability, utility misapplication, and decision incompleteness, and provides empirical evidence that these capabilities can be improved through targeted SFT.

Critical strengths
  1. Well-designed decomposition of decision-making failures.

A major strength is the separation of belief distortion from utility misapplication. By keeping the evidence fixed while varying the utilities and their linguistic framing, the paper can distinguish whether a model changes its belief when the stakes change from whether it simply fails to apply the stated utilities to its own reported belief.

  1. Systematic evaluation and positive results.

The experimental design goes beyond standard in-distribution evaluation by holding out entire application domains, individual framings, and even whole structural classes of payoff descriptions. This makes the claim that SFT teaches a decision procedure rather than merely memorizing particular templates more credible.

Critical weaknesses
  1. The benchmark is substantially simpler than the broader notion of coherent decision-making suggested by the paper.

The experimental setting is restricted to binary decisions with asymmetric false-positive and false-negative costs, where the optimal decision reduces to a single probability threshold. Although this is a useful controlled starting point, it leaves open whether the learned behavior extends to multi-action decisions, multiple possible outcomes, non-linear utility functions, correlated uncertainties, or settings where utilities cannot be reduced to two error costs. The authors acknowledge this limitation themselves, but it is important because the paper's framing around “coherent decision-makers” is broader than what is directly established by the benchmark.

  1. The observed framing generalization is still limited, especially for unseen payoff structures.

Although the paper demonstrates meaningful transfer, the full-class framing holdout results remain considerably worse than the seen-framing results, with the authors reporting belief-conditional decision errors of 0.16 to 0.43 for Direct SFT in this setting. Moreover, CoT SFT can perform worse than the corresponding base CoT model on some held-out full-class framings. This suggests that the learned procedure may still depend substantially on the structural forms encountered during training, which somewhat tempers the stronger claim that the model learns a generally reusable decision procedure.

Questions for authors
  1. How sensitive are the results to the choice of the five utility pairs and the corresponding probability thresholds? Would the conclusions remain similar if the benchmark included a denser and more continuous range of thresholds rather than the five fixed values?

  2. How does the approach behave when the evidence itself is ambiguous or contradictory, or when the model's parametric knowledge conflicts with the external evidence?

  3. Decision making is a widely studied topic, and I don't see the method's novelty very clearly (The related work section doesn't mention similar decision-making approaches neither). Could the authors elaborate on this?

  4. The experiments are done on small to moderate sized LLMs (the largest is 12B). For frontier LLMs, does the problem of decision-making coherency still exists?

Reviewer_Fq9LOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This study explores how fine-tuning enables small language models to make decisions based on given utility while maintaining their original beliefs, and to identify incomplete problems, reject decisions, or perform routing when utility information is missing.

Critical strengths

This manuscript contains formatting errors; I recommend a desk rejection.

Critical weaknesses

This manuscript contains formatting errors; I recommend a desk rejection.

Questions for authors

This manuscript contains formatting errors; I recommend a desk rejection.

Reviewer_MrmCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains small language models to preserve independently elicited beliefs and choose actions consistent with supplied utilities. It separates belief distortion from decision inconsistency and tests transfer across domains and decision framings. A second task withholds utilities and combines a decidability gate with a complete-information specialist.

Critical strengths

Separating belief distortion from decisions inconsistent with the belief stated in the same response is useful. Training on model-relative beliefs also keeps the study focused on coherence rather than conflating it with calibration.

The framing and domain holdouts reveal meaningful limits. Direct SFT improves broadly, while CoT SFT can underperform base CoT on unseen framing classes. Belief preservation is more sensitive to unseen domains. These distinctions are more informative than reporting an overall SFT gain.

Critical weaknesses

The claimed length-matching control is contradicted by Appendix B.4. Section 3.4 and the Discussion say withheld costs are replaced with equally long irrelevant text. All 16 documented replacements are shorter than their cost-bearing counterparts. The intervention false-positive pair has 19 versus 9 words and insurance has 21 versus 10. Unless the actual prompts contain undocumented padding, complete and incomplete items retain a systematic length cue. This weakens the claim that high gate accuracy demonstrates semantic recognition of missing utility information, and therefore also affects interpretation of the routed system.

Section 3.5 justifies Equation 5 by saying independent distortion and sampling noise add in squares. That does not generally hold for mean absolute errors. The caveat that the statistic cannot recover true distortion is appropriate, but does not justify the transformation. Raw distortion and empirical noise measurements are needed to assess the belief-stability comparisons.

Questions for authors

Were the evaluated prompts actually length-matched through a step missing from Appendix B.4? Please report prompt lengths by conditions and models. If the documented strings were used, do the gate and routing gains survive genuinely length-matched replacements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.