acceptodds
Under review as a conference paper at ICLR 2027

Decide, Don't Decode: Turning Post-Training Language Models into Calibrated, Coherent Decision Models

Abstract

Many systems use language models for bounded decisions, such as routing a ticket or flagging a message, and act on the model's confidence. We develop a post-training method that turns an open generative model into a typed decision model: for a choice, a rating or a yes/no question it returns probabilities over the allowed answers without decoding text, trained to be calibrated and coherent across related questions. We build DECIDE-22, 22 typed decision tasks with 22,773 test items; on it, decoding the answer with a stated confidence is no more accurate than reading these probabilities, costs several times more, and yields a confidence that barely separates right from wrong answers (AUROC 0.59, and 0.69 after reasoning, against 0.76). Tracing 26 checkpoints of six model families through every post-training stage they publish, we find that preference optimization sharpens the supervised readout into over-confidence at unchanged accuracy, almost exactly as a temperature would ( 0.92–0.95 in three pipelines). Fine-tuning the readout with a proper scoring rule reverses this sharpening and improves accuracy and calibration, but increases contradictions across logically related questions. We address this with a penalty based on the de Finetti sure loss of each question and its automatically derived siblings. The penalty needs no additional labels and reduces incoherence tenfold without a measurable loss in accuracy. A 9B open model trained this way reaches 76.0% accuracy and a calibration error of 0.054, against 73.3% and 0.113 for a commercial decision API as served (0.068 when given the same calibration recipe), with under a third of its incoherence; at 4B it also beats a concurrent open decision model trained on six times the data. The API keeps its lead on the six held-out tasks, most of it from a single rating scale. We release the benchmark, code and trained models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Schema-defined decision models, System-1 reasoning, Human-anchored distillation, Decision-set invariance, Probabilistic calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BcEnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for fine-tuning LLMs for calibrated decision-making over a discrete set of options. They propose a coherence loss to enforce consistent probabilities across related questions. Experiments are performed across the Decide-22 benchmark.

Critical strengths

Calibration is an important topic in reliable AI deployments. The community would benefit from more focus on LLMs as decision makers.

Critical weaknesses

I find that this paper does not distinguish its contributions from the extensive literature on LLM classification and calibration (e.g., see Section 8 in https://arxiv.org/abs/2602.08145 for extensive background). Text classification is not new; intent classification and toxicity detection have been prevalent for a long time in both research and industry. Fine-tuning for classification typically uses a proper scoring rule, cross-entropy. The paper begins with the claim “A growing share of calls to language models do not ask for text”; this should be cited or further substantiated.

The effects of post-training on calibration are well known (e.g., see https://arxiv.org/abs/2507.16806), and I am not convinced that this study adds substantially to that understanding.

Given the centering of the Decide-22 benchmark as a contribution of this work, it would be good to hear more motivation for why this collection of datasets is particularly meaningful.

The method is a combination of well-known techniques for fine-tuning and recalibrating LLM classifiers, plus a coherence penalty. The empirical effectiveness of the coherence penalty is not substantiated: (1) Table 1 shows no significant gains in accuracy or ECE (2) Comparisons are not made to obvious baselines, like using the additional data for fine-tuning.

Questions for authors

How is this setting different from typical classification with text inputs?

6 comments
AuthorsOpenReview ↗

The paper's first sentence needs a source (Reviewer BcEn)

We agree the opening sentence needed a source, and it is much easier to support now than at submission. The revised first paragraph cites three documented uses of language models for closed-set decisions: labeling data (Gilardi et al., 2023), screening conversations for unsafe content (Inan et al., 2023), and judging other models' answers (Zheng et al., 2023). It also cites five products built specifically for such calls, each returning probabilities over a given set of answers rather than text, all released since September 2026: TypeSafe's Jev, Together AI's Tev1, OpenAI's Decisions API, Perplexity's Decisions API, and Google's Gemini Decisions API. Smaller companies have followed, for example, Liquid AI with its open-weight d1 models. OpenAI describes the uses as classifying content, routing requests, and prioritizing work, and Google uses its own as a query router inside its products. Since we did not measure a share of calls, the sentence now says "many calls" instead of "a growing share".

AuthorsOpenReview ↗

"The effectiveness of the coherence penalty is not substantiated" (Reviewer BcEn):

Reviewer said: (1) Table 1 shows no significant gains in accuracy or ECE. (2) There's no comparison with obvious baselines, like using the additional data for fine-tuning.

"The effectiveness of the coherence penalty is not substantiated." On (1): the penalty isn't meant to raise accuracy or ECE. It targets incoherence, which it cuts tenfold (sure loss 0.291 → 0.024 at 9B), and Table 1 shows it does so at no cost in accuracy or calibration. On (2): we ran the baselines (details in our reply to yNja). Using the additional related questions as labelled training data also removes the incoherence (sure loss 0.034 at 9B), and so do a simple consistency loss (0.022) and the penalty (0.024). Four times as much ordinary labelled data does not (0.311). So the gain comes from training on the question families, not from the projection itself, and the revision says this plainly.

AuthorsOpenReview ↗

Novelty against classification and calibration work; post-training effects are known (Reviewer BcEn):

Reviewer said: ● The paper doesn't separate its contributions from the large literature on LLM classification and calibration (survey arXiv 2602.08145, Section 8). Text classification isn't new, and fine-tuning for classification already uses a proper scoring rule. ● The effects of post-training on calibration are well known (arXiv 2507.16806). ● The method is a combination of well-known techniques plus a coherence penalty. ● Question: how is this setting different from typical classification with text inputs?

How is this different from typical text classification? A text classifier has a fixed label set and is trained for one task. Here one model answers typed questions whose answer options arrive in the prompt at inference time: 22 tasks, three question types, and between 2 and 151 options. Six of the tasks are never used in fine-tuning, and accuracy on them still rises (74.5 → 79.9 at 9B). Most importantly, a classifier has no notion of coherence: whether its answers to logically related questions about the same text could all be true together (an option and the statement it implies, a statement and its negation, a score and a threshold). That property is what this paper is about. We agree that the fine-tuning loss itself is the standard one, cross-entropy as a proper scoring rule, and the paper doesn't claim it as new.

What the paper adds to the classification and calibration literature. The post-hoc methods in the survey you cite (Section 8.2) correct each answer on its own. Our findings are about answers across questions:

  1. Decision fine-tuning of every kind we tried (on the readout, generative SFT, or with four times the data) makes a model's answers to related questions incoherent. At 9B, sure loss goes from 0.141 untuned to 0.23–0.31.

  2. Recalibration can't fix it. We fitted temperature, Platt and contextual-calibration corrections directly on the test families to minimise incoherence, an optimistic bound. Any correction that keeps the model calibrated removes only 10–19% of the incoherence. Going further works only by flattening every answer, which raises ECE four- to six-fold and costs 3–9 points of accuracy.

  3. Training on the question families removes it about tenfold (0.291 → 0.024 at 9B), on held-out tasks too, without costing accuracy or calibration. What incoherence is left now sits where people disagree: on ChaosNLI, the correlation between sure loss and human disagreement goes from −0.35 to +0.25 (Spearman, 9B), and sure loss starts to flag wrong answers (AUROC 0.44–0.49 → 0.57–0.71). Consistency training in general has prior work, which the paper cites (logic-rule losses, paraphrase-consistency losses, inference-time repair). What is new is showing that decision fine-tuning creates the problem, that recalibration can't remove it, and that family training does.

  4. A benchmark that measures all of this (below).

"The effects of post-training on calibration are well known." That post-training hurts calibration is known, and the paper says so ("over-confidence after alignment is known"), citing this line of work: Kadavath et al. (2022) found that one temperature largely repairs an RLHF model's calibration, and Xiao et al. (2025) show ECE rising from SFT to DPO in Tülu 3. The revision also cites the paper you point to (Damani et al., 2025), which finds that RL with binary rewards hurts the confidence a model states after reasoning. Our question is a different one: what each stage of post-training does to a model's decisions. We measure the probabilities read in one forward pass, at every released stage of three open pipelines, with paired intervals, and this adds three things. The over-confidence enters at the preference step (ECE at T = 1 rises 0.03–0.08), and RLVR afterwards leaves it where it was (−0.009 and +0.010); this fits with Damani et al., since the confidence a model states after reasoning and the one-pass readout are different outputs. The change is almost a pure rescaling, so recalibration removes it and discrimination doesn't move. And readout fine-tuning undoes it: started from the SFT checkpoint or from the preference-optimized one, it lands in the same place. This sets up the paper's main result. The calibration damage from post-training can be undone by recalibration; the incoherence that decision fine-tuning creates cannot (recalibration removes only 10–19% of it), and training on question families removes it tenfold.

Reviewer_yNjaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies LLM to probabilistic decision-making with limited options, directly reading the answer probabilities without generating text, and constructs DECIDE-22 for evaluation. The authors analyze the impact of different post-training stages on calibration, and then fine-tune it by combining appropriate scoring rules with penalties based on probabilistic coherence set projection, making the model's answers to the original question and its related questions more consistent.

Critical strengths

(1) The objective is clearly defined, and the experiments cover multiple models, post-training stages, and some multi-random seed settings. In Table 1, the 9B model, while maintaining essentially the same accuracy, reduced the coherence violation index from 0.291 to 0.024, showing significant improvement.

(2) The comparison is relatively comprehensive: the same calibration process was applied to the commercial API, and the transfer of untrained tasks and logical relationships was checked; negative results such as scoring scale degradation and out-of-distribution threshold failure were also reported.

Critical weaknesses

(1) The unique value of projection penalties has not been fully isolated. This method additionally evaluates related questions during training, while the original labels themselves can derive labels for many related questions.

(2) The paper lacks a matching comparison of "directly supervised training on the same set of related problems" and simple consistency loss, so it is unclear how much of the benefit comes from the projected target and how much comes from exposure to related problems during training. The existing Tev1 uses different training data or configurations than earlier decision head schemes, which cannot resolve this attribution issue.

Questions for authors

(1) Under similar training budgets, can directly supervising the same related problems or using simple consistency loss achieve similar results?

(2) Does the high R² of temperature scaling still hold after task-balanced statistics, or is it partially affected by the weights of multi-option tasks?

(3) Given the limited transfer of unknown relation types and the potential failure of out-of-distribution thresholds, how should the applicability of this decision model be defined?

4 comments
AuthorsOpenReview ↗

The projection penalty isn't isolated (Reviewer yNja, weaknesses 1–2 and Q1):

Reviewer said: ● The method also evaluates related questions during training, and the original labels can derive labels for many of them. ● There's no matched comparison with directly supervising the same related questions, or with a simple consistency loss. So it's unclear how much comes from the projected target and how much from seeing related questions. Tev1 uses different data and settings, so it can't settle this. ● Q1: under similar budgets, do direct supervision or a simple consistency loss do as well?

Thank you for the review. Your first question asked for exactly the right control, and we ran it.

Q1 (and weaknesses 1–2): the projection or the related questions? The related questions. The paper compared the penalty only with fine-tuning on the original questions, because the projected target comes with a guarantee (it never raises the Brier score) and we expected it to be what mattered. We have now run the two controls you describe at the same budget, plus a third with four times as much ordinary labelled data, with the paper's settings (Qwen3.5-2B and 9B, three seeds each). The two controls see exactly the same related questions and forward passes as the penalty, and both remove the incoherence. At 9B the sure loss is 0.022 with a simple consistency loss and 0.034 with derived labels, against 0.024 with the projection penalty and 0.291 with fine-tuning on the original questions alone; at 2B it is 0.026, 0.041 and 0.033 against 0.335. Accuracy and ECE stay the same across these (9B: 75.9–76.1%, ECE 0.053–0.056), and the held-out tasks show the same pattern (9B sure loss 0.032–0.039 against 0.223). Four times as much ordinary data raises accuracy by 1.5 points at 9B but leaves the model as incoherent as before (0.311).

So what removes the incoherence is training on each question together with its related questions; the projection is one good way to do that, not the only one. This makes the main result broader, not different: ordinary decision fine-tuning makes a model's answers incoherent, more data and recalibration don't fix it, and training on the question families does, about tenfold, at no cost in accuracy or calibration. The revision says this in the abstract, the contributions and Section 5, and adds the controls to the appendix. We also agree that Tev1 could not establish this attribution because its data and experimental setup differ. The new experiments directly address that gap and provide a stronger, more complete explanation of our findings.

AuthorsOpenReview ↗

Does the R² hold with every task weighted equally? (Reviewer yNja, Q2):

Reviewer said: does the high R² of the temperature analysis survive task-balanced statistics, or is it driven by the tasks with many options?

Q2. Does the high R² survive task-balanced statistics? Yes. With every task weighted equally, R² is 0.906 for Tülu 3, 0.912 for SmolLM3 and 0.946 for OLMo 2 (0.917, 0.928 and 0.947 in the paper), and dropping the tasks with 28 or more options gives 0.932, 0.920 and 0.949. Those tasks hold 88% of the regression points but don't drive the result. The limit the paper states also holds task-balanced: one temperature removes only a third to a half of the change in the answer distributions. The revision adds this to the appendix.

AuthorsOpenReview ↗

How should applicability be defined? (Reviewer yNja, Q3):

Reviewer said: given the limited transfer to unseen relation types and the possible failure of thresholds out of distribution, how should the applicability of this decision model be defined?

Q3. How should applicability be defined? The revision adds a scope paragraph. The method applies to closed-set decisions about a text where the answer set is given in the question: a choice among K named options (we test up to K = 151), an ordered scale, or a yes/no statement. Coherence is trained for the relations the families encode (an option and its statement, negation, thresholds). It transfers only partly to relation types never trained on, and a threshold or rule stated in the question can fail out of distribution, as the paper reports. Accuracy gains are largest on the tasks trained for. On ten public benchmarks never used in training, fine-tuning still helps (+3.6, +4.7 and +7.2 points over the untuned 9B, 4B and 2B models), and the projection penalty neither helps nor hurts (+0.0 points [−0.3, +0.3] over five matched pairs), but the commercial API leads there by 7–9 points. A deployment should refit the recipe on its own validation data.

Thanks for the additional experiments to clarify my concerns. I would like to maintain my positive score.

Reviewer_UzeCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Authors fine-tune a set of autoregressive models with a new loss that combines a SFT term depending only on the token that identifies the true class and a coherence penalty to ensure coherence across those labels. They compare cases that include models that produce generations and decisions.

Critical strengths

The paper provides a large number of models and tasks and enough seeds for the runs. This gives strength to the results, but the conclusions extracted from them do not correspond, to my understanding, to a valid answer to the questions raised in the introduction. In addition, the authors acknowledge using AI to generate the content of the paper and take responsibility for it.

Critical weaknesses

The paper presents several methodological problems and writing inconsistencies. Following, there are some examples of each. I do not consider writing inconsistencies severe, but they clearly evidence that, in my opinion, the paper did not have enough human supervision to be accepted.

  • Examples of writing inconsistencies: Lines 258-259 of section 4 state “the answers keep their order and become sharper, which is why accuracy barely moves”, however accuracy is not shown in any plots of this or the previous section. Line 137 says “so that K up to 151 still fits one step” without clarifying that 151 is the maximum number of supported categories. ss and qq in equation 1 are not clearly defined (although I admit it is understood that they are the state and the question). More importantly, a lot of architecture families are mentioned at the beginning of section 4 but some of them are missing in the plots of that section and the appendices mentioned within that section.

  • Examples of methodological issues: In section 2, authors do not explain why the readout calibration is done by an average followed by Temp Scaling for the choice answers and bias-shift for the yes/no questions, which is a very non-standard procedure. Importantly, Figure 2 shows the value of the temperature, which is not a valid measure of calibration (like ECE or RelCalError). In addition, showing the correlation between log-scores of each stage does not fully show that some stages could be replaced by a temperature if calibration is not measured. Moreover, even if ECE=0 in both cases (which it may not be the case if there is another type of miscalibration that Temp Scaling does not correct), it would not necessarily be replaced by a Temperature, since calibration does not measure discrimination.

Questions for authors
  • The paper introduction claims to answer the two main questions raised by the paper (lines 76-78): “What does each stage of post-training do to a model’s decisions, as opposed to its generations, and can a model then be post-trained for decisions directly?” What is the exact distinction between “decision” and “generation” and how “trained for decisions directly” differ from traditional SFT?
  • Is SFT the decisions better than the generations or is it that the SFT model was trained on additional task-specific data?
  • Given that the time per decision is better for the readout than for the Jev model, I suspect that a SFT only in the tasks of interest is better than the Jev API in terms of performance and efficiency for these tasks. Please, can you confirm with a citation whether there is a better model than the Jev for this task in terms of the used metrics?
11 comments
AuthorsOpenReview ↗

Accuracy "not shown" (Reviewer UzeC):

Accuracy is in Figure 2, but we agree it is easy to miss: the number under each stage's name is that checkpoint's accuracy on Decide-22. Table 3 (Appendix C) lists accuracy for all 26 checkpoints. The preference step changes accuracy by +0.4 points in Tülu 3, +0.3 in OLMo 2, and +1.8 in SmolLM3. "Barely moves" was too strong for SmolLM3, so the revised text now gives these numbers and points to Figure 2 and Table 3 (Section 4, marked in blue).

AuthorsOpenReview ↗

"K up to 151" and the symbols in Eq. 1 (Reviewer UzeC):

151 is not a limit of the method. It is the largest answer set in Decide-22: CLINC150's 150 intents, plus out-of-scope (see Section 3 and Table 2). We now say so where the number first appears. Larger answer sets work the same way, as long as the tokenizer has enough single-token codes. Eq. 1 now defines every symbol: s is the state, q the question, π_θ the model's next-token distribution, t(a_k) the token of answer a_k, and T the temperature fitted by the recipe (T = 1 is the raw readout).

AuthorsOpenReview ↗

Model families missing from the Section 4 figures (Reviewer UzeC):

You are right that the Section 4 figures do not show every family, and the text should have said so. Figure 2 covers the three families that publish their intermediate stages (Tülu 3, OLMo 2, SmolLM3). Qwen3.5, Qwen3, and Gemma-4 publish only base and fully post-trained checkpoints, so there is no stage-by-stage trace to plot. All 26 checkpoints from all six families are in Table 3 (Appendix C), with accuracy, ECE, fitted temperatures, sure loss, and order flips. Section 4 now says this. Figure 8 left out OLMo 2. Its values were in Figure 7, and Figure 8's caption now states them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (all reviewers):

We thank the reviewers. We ran the experiments their questions asked for. Every number in our replies comes from finished runs, and the changes in the revised PDF are marked in blue.

  1. What removes the incoherence (yNja, BcEn). Matched controls at the same budget show that training on each question together with its automatically derived related questions removes the incoherence that decision fine-tuning creates, about tenfold and on held-out tasks too. This holds whether the training signal is our projection penalty, a simple consistency loss or derived labels. Four times more ordinary data doesn't fix it, and neither does post-hoc recalibration. The revision states the contribution this way, with the sure loss as the measure.

  2. Calibration measured directly (UzeC). ECE, Brier and AUROC for every post-training checkpoint, at T = 1 and after recalibration: the preference step adds over-confidence, recalibration removes it, and discrimination is unchanged.

  3. Decisions vs generations (UzeC). Generative SFT on the same data, evaluated by reading and by decoding: on the trained tasks the gain is mostly the data; decoding costs 3–4 points there and 12–14 on held-out tasks.

  4. Robustness (yNja). The temperature-equivalence R² holds with every task weighted equally (0.91–0.95).

  5. Scope (yNja, UzeC). On ten public benchmarks never used in training, fine-tuning still helps and the penalty doesn't change accuracy. The commercial API leads there, and we scope our claims to the trained tasks.

  6. Presentation (UzeC, BcEn). The missing header is restored, the recipe is explained with a per-step ablation, Section 4 now points to accuracy and says where each model family is shown, the opening claim is cited, and Decide-22 is motivated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.