acceptodds
Under review as a conference paper at ICLR 2027

Task Calibrated Decoding: Improving Black-Box LLMs with In-Domain Data

Abstract

When using Large Language Models (LLMs) in downstream tasks, labeled in-domain examples provide a natural source of supervision. However, finetuning the underlying LLM can be costly or intractable, particularly when access is restricted to only sampling responses in a black-box manner. We propose Task-Calibrated Decoding, a lightweight procedure that uses labeled examples to improve decoding in a frozen, black-box LLM. We represent sampled free-form responses as a distribution over their task-relevant semantics, such as numerical ratings or class labels. Using in-domain labels, we calibrate the full distribution and apply Minimum Bayes Risk Decoding to select the response minimizing expected task loss. The procedure requires no access to model weights or token probabilities, and is provably optimal among decoding strategies based on the LLM’s output distribution under ideal recalibration. Empirically, it improves generation quality across baselines for different tasks and models, and we present qualitative examples showing how it reduces systematic biases. Our results demonstrate that task-calibrated decoding is a theoretically motivated, lightweight, and broadly applicable strategy to exploit in-domain supervision by adapting downstream decisions while keeping the underlying model fixed

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Decoding, Domain Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_b3prOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Task-Calibrated Decoding (TCD), a black-box adaptation framework that uses labeled in-domain data to improve downstream decisions without updating the underlying LLM. It maps multiple sampled responses into a task-specific semantic distribution, fits a Dirichlet calibration map, and applies Minimum Bayes Risk (MBR) decoding under the task loss. The paper provides an idealized optimality justification and evaluates the framework across several models and task settings.

Critical strengths
  1. The framework addresses a practical setting where model weights and token probabilities are unavailable, but sampled responses and in-domain labels can be obtained.
  2. Separating distribution estimation from task-specific decision-making is conceptually clear. Retaining the calibrated distribution potentially allows different decision costs to be accommodated without refitting the calibrator.
  3. The paper includes comparisons with several post-processing methods and LoRA, together with ablations on sampling budget, calibration-set size, and prompt variations.
Critical weaknesses
  1. Limited evidence of additional methodological value. TCD combines existing Dirichlet calibration with standard MBR decoding. In Table 1, it exactly matches the Classifier baseline on both classification tasks. For other tasks, comparisons with simple predictors trained directly for the task loss would better establish the benefit of full-distribution calibration.

  2. The lightweight claim needs qualification. TCD samples 20 responses per query and may require auxiliary models for semantic mapping, whereas ICL generates one response. End-to-end latency, token costs, and budget-matched comparisons are needed to assess its practical advantage.

  3. The evaluation is narrower than presented. The MMLU results only cover moral_scenarios, and most experiments use in-distribution cross-validation. Robustness to distribution shifts and changes in label frequencies remains unclear.

  4. Some gains come from always abstaining. Appendix A.2 reports this behavior for several models on SimpleQA. Coverage–accuracy curves and simple abstention-threshold baselines would clarify whether TCD offers useful selective answering.

Questions for authors

See weakness

8 comments
AuthorsOpenReview ↗

Rebuttal b3pr

Thank you for taking the time to review the paper. Overall we are glad that you see the relevance of our method and we think you raised some good points which we address below (new empirical results are present in Figshare)

W1: We really appreciate the suggestion to compare with a classifier that is trained on the task-specific loss, as we agree that this can make the paper significantly stronger. In particular, we train a linear classifier similar to TCD to minimize the empirical task loss. In specific, we train the following objective with L2 regularization:

\begin{equation} \min_{W,b} E_{(p,y)} \sum_{a \in \mathcal{A}} \operatorname{softmax}(Wp+b)_a \ell(a,y) \end{equation}

where pp is the input predictive distribution, A\mathcal{A} is the action space, softmax⁡(Wp+b)a\operatorname{softmax}(Wp+b)_a denotes the probability assigned to action aa, and ℓ(a,y)\ell(a,y) is the task-specific loss of taking action aa when the ground-truth outcome is yy. At inference time, we select the action with the highest predicted probability:

\begin{equation} \hat{a}(p) = \arg\max_{a \in \mathcal{A}} \operatorname{softmax}(Wp+b)_a. \end{equation}

While a strong baseline, we find it to underperform TCD, see Table 1 (Task-Classifier).

Furthermore, this baseline supports decision-making only for this fixed loss, whereas TCD reuses the same calibrated distribution across different losses without retraining. Our new ablation (Figure 3) explicitly visualizes this: across different thresholds tt in the BAS Metric (each defining its own unique loss), TCD maintains strong performance while the baselines deteriorate. This is again distinct from the newly added task-dependent classifiers, which are tailored to a single loss (t=0.25t=0.25) and are also shown to deteriorate in Figure 3.

W2: We agree that an exploration of different efficiency-performance trade-offs for both TCD and the baselines is insightful. We first present, in Table 1, ICL using 20 samples and MBR as the decision criterion, and find that TCD continues to outperform at this compute budget which matches TCD. For a more detailed efficiency-performance analysis, we present Figure 1, which shows latency versus task loss. In particular, we again evaluate ICL with additional samples to allow larger inference budgets. In our evaluation, TCD Pareto-dominates ICL, also at relatively low sampling budgets. Therefore, TCD enables a practitioner to trade off inference latency for task performance by adjusting the number of samples. These measurements include both LLM sampling and auxiliary semantic mapping (decomposition in Figure 2). The decomposition also shows that auxiliary mapping costs are rather low and the post-hoc decision procedure is negligible (microseconds).

W3: Our MMLU evaluation indeed only uses moral_scenarios, but we believe that our broad range of tasks and datasets sufficiently shows the practical efficacy of TCD. Should you think that another MMLU subset would strengthen the evaluation, we can also run experiments on this.

The discussion regarding distribution shifts is highly relevant: Like any other fine-tuning or post-hoc method, we rely on the labeled data to represent the data distribution at inference. Consequently, we do not expect the calibrator of TCD to be robust to strong distribution shifts. This is an important limitation that we will acknowledge explicitly in our revision (see updated manuscript). Addressing it, however, is beyond the scope of this work.

W4: Indeed, for weaker models, the performance gains in answer-abstain come from always abstaining. This behavior is consistent with the decision objective: for models with low answer accuracy, abstaining minimizes the expected incurred loss. Here, always abstaining may be the optimal policy, and a loss-aware decision rule should be able to recover this behavior. For stronger models, TCD allows to answer if the model confidence is sufficiently high, indicating that we do not abstain in general.

We also appreciate the suggestion to study different thresholds tt for this answer-abstain problem. As mentioned above (W1), Figure 3 shows that TCD is the only method that maintains performance across different thresholds, whereas the baselines deteriorate. We also provide an accuracy-coverage plot across the 5 CV Folds for TriviaQA, for both calibrated and uncalibrated probabilities in Figure 4, which we find to differ only in their specific thresholds. However, we see this as somewhat orthogonal to TCD, as TCD, in this case, is optimized with respect to a decision under the BAS metric (specified for specific tt) and produces not a score that can be thresholded, but instead directly the decision of whether to answer or abstain.

Thank you for your suggestions. We hope that the additional experiments help to strengthen the justification and performance of TCD. If there are further points, we are happy to discuss!

Thank you for the response. My concerns W1–W4 remain insufficiently resolved based on the manuscript currently available.

  1. W1: The proposed Task-Classifier comparison is relevant, but its results are not included in the provided PDF. Please report the comparison with sufficient training details to establish the additional value of TCD; its Figure 3 differs from the experiment described in the response.

  2. W2: The claimed latency–performance analysis is also missing. Figure 1 in the manuscript is a pipeline schematic, not the plot described in the rebuttal. Please provide the measured results and clarify whether all sampling and auxiliary mapping costs are included.

  3. W3: Acknowledging distribution shift as a limitation clarifies the scope but does not address the empirical concern. The evaluation remains restricted to one MMLU subset and predominantly in-distribution settings.

  4. W4: Explaining why always abstaining can minimize the specified loss does not demonstrate useful selective answering. The new threshold and accuracy–coverage analyses are needed to assess this point, but they are not present in the provided PDF.

Please provide a direct link to the additional results mentioned on Figures and incorporate them into a revised PDF with clear figure references. Until these results can be examined, I cannot consider these concerns resolved.

Thank you for engaging with us in the discussion.

There seems to have been a misunderstanding: In our response, we provided a link to our new results (which are in figshare) in our first paragraph (if you click on figshare you should get to that) to make it easier for the reviewers to find the results relevant to the rebuttal. Here is the link again: https://figshare.com/s/3ba788b42ab03bed78d1

We apologise that we didn’t make the link clearer.

W3: Regarding evaluation on other MMLU datasets, running inference for all models on the entire data is computationally infeasible. Do you have a suggestion as to which subset would be interesting for you, so we can supply the numbers?

Regarding distribution shifts: Again, we would expect that all methods, including TCD, are vulnerable to out-of-distribution data, so we see little additional value in empirically showing this. However, if you believe that such an experiment would be interesting to see, we are would also be very grateful for pointers toward relevant settings.

W4: Again, refer to the Figshare document for the coverage-accuracy plots.

TCD’s primary goal is to optimize the downstream loss. Importantly, the BAS metric does not optimize for a specific trade-off between coverage and accuracy. Instead, for example, at threshold t=0.25t=0.25: a correct answer earns +1+1, an incorrect answer earns −1/3-1/3, and abstaining earns 00. Under this scenario, we can compare different strategies with different coverage and accuracy metrics in terms of their BAS metric:

For 100 questions:

StrategyCorrectIncorrectAbstainCoverageAccuracy on answered questionsMean utility
Answer everything20800100%20%−0.067-0.067
Answer only 202188020%10%−0.040-0.040
Always abstain001000%Undefined00

The second strategy has lower coverage and lower selective accuracy, yet better BAS-derived utility: it incurs fewer penalties overall. The key is that both answering strategies have accuracy below the break-even threshold of 25%25\%. Answering fewer questions can therefore improve utility even when the remaining answers are less accurate.

Thus, TCD in conjunction with the BAS metric is not guaranteed to improve either accuracy or coverage. If a practitioner values these metrics, the correct approach is to define a suitable downstream loss that penalizes for example low accuracy or coverage, and then run TCD with respect to this metric. If you are aware of a specific suitable loss function, we are happy to provide this experiment as well.

Reviewer_feUpOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes improving task-level decisions from an LLM without requiring weights or token probabilities. It delineates the following workflow:

  1. Generating multiple responses and mapping them to task outcomes - ratings, class labels, answer set membership.
  2. Using labeled in-domain examples to calibrate the distribution, correcting the empirical distribution
  3. Applying MBR decoding and choosing a loss-aware decision The experiments demonstrate improvements over uncalibrated decoding over 5 models and 8 task settings.
Critical strengths
  1. The method is simple and practical. It allows the use of frontier LLMs as black boxes, eschews the need for finetuning, and requires only ~30 calibration parameters for a five-outcome task.
  2. The evaluation includes useful comparisons and ablations. The paper reports 5-fold CV, including a classifier over the same fitted calibration map, compared against decision calibration and LoRA, and takes into account concerns such as prompt changes and sampling budgets.
  3. Cases where competing methods perform better are acknowledged.
Critical weaknesses

None

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Rebuttal feUp

We are happy that you see practical value in our method, especially as you emphasize its use for frontier LLMs, where fine-tuning might not be possible due to their black-box nature. Similarly, we appreciate your emphasis on our broad range of ablations and, importantly, our acknowledgment of limitations where necessary, as we see this as a crucial part of scientific rigor.

Reviewer_9t7KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Task-Calibrated Decoding (TCD), which uses labeled in-domain data to improve the decisions of a black-box LLM with three steps. First, it samples multiple responses, converts them into task outcomes such as scores, classes, or actions, and estimates their probabilities from their frequencies. Second, it trains a Dirichlet calibrator on labeled data to adjust these probabilities. Third, it uses Minimum Bayes Risk (MBR) decoding to choose the outcome with the lowest expected task loss. The authors show that, with an ideal calibration map, this rule minimizes expected loss among all methods that rely only on the original task-outcome distribution. Experiments across five models and eight task settings—including scoring, classification, tool-use decisions, answering or abstaining, and multi-answer prediction—show that calibration generally improves performance over aggregating samples without calibration.

Critical strengths
  • TCD only trains a small calibrator and keeps the LLM frozen. For systems that already use multiple samples, it adds little computational cost.

  • The authors evaluate TCD across multiple tasks and models, including both open-weight and closed-source models, demonstrating its effectiveness in different settings.

Critical weaknesses
  • Although the calibrator is small, the method typically requires 20 samples per question. Some tasks also use an auxiliary model to extract answers. This adds computation and cost, making the method less suitable for applications that require a single model call or low latency.

  • The calibrator’s structure depends on the number of candidates. For example, a calibrator trained for five candidates cannot be directly applied to a task with ten candidates. Even with the same number of candidates, changing the model or task may require revalidation or retraining.

  • The method has limited novelty, as it mainly combines existing semantic mapping, Dirichlet calibration, and MBR decoding. For standard classification tasks, TCD is equivalent to training a classifier on vote proportions, and the paper’s Classifier baseline achieves exactly the same results.

Questions for authors
  • The paper lacks a comparison of computational costs. ICL generates one final answer, while TCD uses multiple samples. The results show performance differences, but do not establish that TCD is more effective under the same computational budget.

  • The reported standard deviations suggest substantial variability in some settings. The current results do not establish whether TCD significantly outperforms the baselines. The authors should provide statistical significance tests.

I would be happy to raise my rating if the authors adequately address the concerns in the Weaknesses and Questions.

2 comments
AuthorsOpenReview ↗

Rebuttal 9t7K

Thank you for your review and suggestions. Below we address your concerns with empirical evidence which we agree will make the paper overall stronger. New empirical results can be found in Figshare

W1 and Q1: To make a fairer comparison to ICL, we now include ICL with 20 samples and the MBR criterion for the final decision to match the computational budget of TCD. Results are shown in Table 1, and TCD continues to outperform at this compute-matched budget. To explore the efficiency-performance trade-off more we provide a more detailed analysis in Figures 1 & 2. In Figure 1, we visualize performance versus latency across different computational budgets by varying the number of Monte Carlo samples. We can see that, across latency-matched budgets, TCD dominates the Pareto frontier, allowing practitioners to balance inference cost against performance by adjusting the number of Monte Carlo samples. Similar evidence from MC samples in the original paper (Figure 2a original paper) shows that with a small number of samples, we can already achieve performance gains, reemphasizing that if a user cares about compute & latency, TCD can still provide benefits. Lastly, in Figure 2, we detail how these costs are composed over sampling and mapping to discrete outcomes. Auxiliary model costs are comparably low, and the post-hoc decision procedure is negligible, taking microseconds, making the aforementioned MC sampling budget the key parameter. Should you miss a comparison or another angle of evaluation, we are happy to provide more empirical data.

W2: We agree that the calibrator operates on a fixed outcome space and model. However, refitting the calibrator is computationally inexpensive and hence can be easily done should the outcome space or model change. Importantly, however, the fixed outcome space does not necessarily restrict TCD to simple problems: our set prediction task (MAQA) demonstrates how structured problems can be decomposed into fixed-outcome decision problems.

W3: We agree that TCD builds on Dirichlet calibration and task-aware MBR decoding. Our main contribution is to connect calibration with MBR decoding through the lens of a task-dependent interpretation of black-box LLM responses. This connection is non-trivial, as the naive implementation of the task-dependent MBR prior work would simply be to train a classifier using the task-specific loss to directly improve downstream decisions. To showcase this, we now include this baseline in Table 1, and find it to be inferior to TCD. Our connection explicitly requires making use of Proposition 3.1, which states that TCD is optimal when achieving a distributionally calibrated distribution, as approximated by Dirichlet calibration. Moreover, as pointed out by b3pr, TCD’s calibrated distribution is reusable across different downstream losses without retraining by separating LLM-based task-level probability from decision-making. Our new ablation (Figure 3) explicitly visualizes this: across different thresholds tt in the BAS Metric (each defining its own unique loss), TCD maintains strong performance while the baselines deteriorate. This is again distinct from the natural extension of task-dependent classifiers, which are tailored to a single loss and are also shown to deteriorate in Figure 3.

That TCD reduces to standard classification under 0-1 loss is by construction, as this is a special case of MBR. However, as demonstrated in Tab. 1., under different losses, the classifier trained on vote proportions (Classifier) underperforms TCD. We also want to highlight that our work enables the study of calibration in LLMs in a task-specific way, which we believe is a practically appealing direction for further research

Q2: In Table 2, we report a statistical evaluation using 1000 resamples of the calibration and test sets with an 90/10 split ratio, refitting TCD for each resample (as the calibrator is computationally inexpensive to refit). We report 95% confidence intervals and p-values for comparing the mean performance of TCD versus the baselines using the Nadeau-Bengio corrected resampled t-test. This establishes statistical significance for the majority of settings in which TCD outperforms at p<0.05. We will include this table in our camera-ready version.

Thank you for these suggestions, and we hope that the additional analysis strengthens the empirical evaluation and helps to clarify TCD’s practical performance-efficiency trade-offs. Should you have further points to discuss, we are happy to engage with them.

Thanks for the responses. I have updated my rating.

Reviewer_CMJ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Task-Calibrated Decoding (TCD), which uses labeled in-domain data to improve LLM decisions without fine-tuning. It samples multiple responses to the same question, maps different phrasings to discrete outcomes such as ratings or classes, and estimates their probabilities from sample frequencies. A small calibrator then adjusts these probabilities, and Minimum Bayes Risk (MBR) decoding selects the outcome with the lowest expected task loss rather than simply choosing the most frequent answer. The main idea is to combine task-level probability calibration with loss-aware decision-making. Experiments across five models and eight task settings, covering rating, classification, tool use, and QA, generally show improvements over uncalibrated MBR and several lightweight baselines.

Critical strengths
  1. The method has clear practical value for black-box models. It requires neither model weights nor token probabilities, using only sampled responses and in-domain labels to fit a small post-processing calibrator. This offers a viable way to adapt models that are accessible only through a generation API. These limited access requirements are a main reason I see practical significance in the work.
  2. The experiments directly test an important part of the design. Beyond evaluating multiple models and tasks, the authors include a Classifier baseline that uses the same fitted calibrator but selects the highest-probability label. TCD performs better on some rating tasks, suggesting that the gains come not only from additional labeled data but also from choosing answers according to the task loss. This comparison is more convincing than comparing only against uncalibrated predictions, and supports the value of combining calibration with MBR.
Critical weaknesses
  1. The methodological contribution appears incremental. TCD mainly adds standard Dirichlet calibration to the task-space MBR framework of [1]. The experiments show that this combination works, but the method itself seems closer to a straightforward extension of existing work.
  2. The applicability to free-form generation remains limited. Although responses can vary in wording, the method ultimately operates on discrete ratings, classes, or answer-or-abstain decisions. The current experiments therefore do not establish improvements in general open-ended text generation.
  3. The cost advantage is not sufficiently established. The calibrator is small, but the main experiments require 20 samples per question, sometimes with auxiliary models for mapping. These costs recur with each query, whereas fine-tuning costs can be amortized. Without matched-budget comparisons and longer-term cost analysis, it is unclear whether the gains justify the additional computation.

[1] https://arxiv.org/abs/1910.12656

Questions for authors

Please see Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal CMJ2

Thank you for your feedback; we appreciate that you see clear practical value in our method and the well-supported experimental evidence for its efficacy. Below we address your points and share all new empirical results in Figshare

W1: We agree that our method builds on existing work. Our main contribution is to connect calibration with MBR decoding through the lens of a task-dependent interpretation of LLM responses. This connection is non-trivial, as a natural implementation of the task-dependent MBR prior work would simply involve training a classifier using the task-specific loss to improve downstream decisions. To showcase this, we now include this additional baseline in Table 1 and find it to be inferior to TCD. Our connection between calibration, task-dependence, and MBR explicitly requires making use of Proposition 3.1: It states that TCD is optimal when achieving a distributionally calibrated distribution, as approximated by Dirichlet calibration. Moreover, as pointed out by b3pr, TCD’s calibrated distribution is reusable across different downstream losses without retraining by separating LLM-based task-level probability from decision-making. Our new ablation (Figure 3) explicitly visualizes this: across different thresholds tt in the BAS Metric (each defining its own unique loss), TCD maintains strong performance while the baselines deteriorate. This is again distinct from the natural extension of task-dependent classifiers, which are tailored to a single loss and are also shown to deteriorate in Figure 3. Beyond that, our work enables the study of calibration in LLMs in a task-specific way, which we believe is a practically appealing direction for further research.

W2: TCD specifically targets LLM-based decision making rather than improving the quality of unconstrained text generation. However, when LLM responses are algorithmically used in downstream decision-making, they often explicitly need to be mapped to an outcome space.

While our experiments focus on a specific set of discrete task outcomes, TCD can also accommodate more open-ended generation whenever a suitable mapping to outcomes is available. For instance, in code generation, programs can be mapped to task-specific outcomes and TCD may be used to calibrate whether to accept or reject a candidate. We do not claim that TCD improves unconstrained text generation without any downstream problem at hand, and we will clearly highlight this more in our revised paper (see updated manuscript, blue additions). However, requiring a discrete outcome space does not automatically restrict TCD to tasks with discrete model outputs. Should you have a suggestion here, we are glad to provide empirical data.

W3: We agree that exploring the performance-efficiency trade-off would provide more insights, and we ran experiments accordingly. We first present in Table 1 ICL using 20 samples as well, with the MBR criterion for the final decision, and find that TCD outperforms. We also provide a more detailed efficiency-performance analysis in Figures 1 & 2. In Figure 1, we visualize performance versus latency across different computational budgets by varying the number of Monte Carlo samples. LoRA responses are trained to omit reasoning and require a single decoding pass (Beam-Search), often being computationally cheaper than TCD, at the cost of performance. In Figure 2, we detail how these costs are composed over sampling and mapping to discrete outcomes. Auxiliary model costs are comparably low, and the post-hoc decision procedure is negligible, as it is in the microseconds.

All other baselines are strictly Pareto-dominated by TCD at different budgets. Therefore, TCD allows a practitioner to realize a specific broad regime of performance-efficiency trade-offs. Further, while LoRA amortizes training costs over many queries, it requires access to model weights, whereas TCD applies to black-box APIs without incurring parameter update costs. We also want to note that LoRA and TCD are not mutually exclusive: TCD can be applied to any fine-tuned LLM, and we emphasize that we do not propose TCD as an alternative to LoRA for open-weight models but as another practical, effective tool for leveraging LLMs for downstream problems.

Thank you for suggesting these points. We hoped that we could clarify TCD’s applicability and justification in practice, also by establishing computational trade-offs. If there are any further points remaining, we are happy to discuss!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.