- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Task-Calibrated Decoding (TCD), a black-box adaptation framework that uses labeled in-domain data to improve downstream decisions without updating the underlying LLM. It maps multiple sampled responses into a task-specific semantic distribution, fits a Dirichlet calibration map, and applies Minimum Bayes Risk (MBR) decoding under the task loss. The paper provides an idealized optimality justification and evaluates the framework across several models and task settings.
Critical strengths
- The framework addresses a practical setting where model weights and token probabilities are unavailable, but sampled responses and in-domain labels can be obtained.
- Separating distribution estimation from task-specific decision-making is conceptually clear. Retaining the calibrated distribution potentially allows different decision costs to be accommodated without refitting the calibrator.
- The paper includes comparisons with several post-processing methods and LoRA, together with ablations on sampling budget, calibration-set size, and prompt variations.
Critical weaknesses
-
Limited evidence of additional methodological value. TCD combines existing Dirichlet calibration with standard MBR decoding. In Table 1, it exactly matches the Classifier baseline on both classification tasks. For other tasks, comparisons with simple predictors trained directly for the task loss would better establish the benefit of full-distribution calibration.
-
The lightweight claim needs qualification. TCD samples 20 responses per query and may require auxiliary models for semantic mapping, whereas ICL generates one response. End-to-end latency, token costs, and budget-matched comparisons are needed to assess its practical advantage.
-
The evaluation is narrower than presented. The MMLU results only cover moral_scenarios, and most experiments use in-distribution cross-validation. Robustness to distribution shifts and changes in label frequencies remains unclear.
-
Some gains come from always abstaining. Appendix A.2 reports this behavior for several models on SimpleQA. Coverage–accuracy curves and simple abstention-threshold baselines would clarify whether TCD offers useful selective answering.
Questions for authors
See weakness
8 comments
Thank you for the response. My concerns W1–W4 remain insufficiently resolved based on the manuscript currently available.
-
W1: The proposed Task-Classifier comparison is relevant, but its results are not included in the provided PDF. Please report the comparison with sufficient training details to establish the additional value of TCD; its Figure 3 differs from the experiment described in the response.
-
W2: The claimed latency–performance analysis is also missing. Figure 1 in the manuscript is a pipeline schematic, not the plot described in the rebuttal. Please provide the measured results and clarify whether all sampling and auxiliary mapping costs are included.
-
W3: Acknowledging distribution shift as a limitation clarifies the scope but does not address the empirical concern. The evaluation remains restricted to one MMLU subset and predominantly in-distribution settings.
-
W4: Explaining why always abstaining can minimize the specified loss does not demonstrate useful selective answering. The new threshold and accuracy–coverage analyses are needed to assess this point, but they are not present in the provided PDF.
Please provide a direct link to the additional results mentioned on Figures and incorporate them into a revised PDF with clear figure references. Until these results can be examined, I cannot consider these concerns resolved.
Thank you for engaging with us in the discussion.
There seems to have been a misunderstanding: In our response, we provided a link to our new results (which are in figshare) in our first paragraph (if you click on figshare you should get to that) to make it easier for the reviewers to find the results relevant to the rebuttal. Here is the link again: https://figshare.com/s/3ba788b42ab03bed78d1
We apologise that we didn’t make the link clearer.
W3: Regarding evaluation on other MMLU datasets, running inference for all models on the entire data is computationally infeasible. Do you have a suggestion as to which subset would be interesting for you, so we can supply the numbers?
Regarding distribution shifts: Again, we would expect that all methods, including TCD, are vulnerable to out-of-distribution data, so we see little additional value in empirically showing this. However, if you believe that such an experiment would be interesting to see, we are would also be very grateful for pointers toward relevant settings.
W4: Again, refer to the Figshare document for the coverage-accuracy plots.
TCD’s primary goal is to optimize the downstream loss. Importantly, the BAS metric does not optimize for a specific trade-off between coverage and accuracy. Instead, for example, at threshold : a correct answer earns , an incorrect answer earns , and abstaining earns . Under this scenario, we can compare different strategies with different coverage and accuracy metrics in terms of their BAS metric:
For 100 questions:
| Strategy | Correct | Incorrect | Abstain | Coverage | Accuracy on answered questions | Mean utility |
|---|---|---|---|---|---|---|
| Answer everything | 20 | 80 | 0 | 100% | 20% | |
| Answer only 20 | 2 | 18 | 80 | 20% | 10% | |
| Always abstain | 0 | 0 | 100 | 0% | Undefined |
The second strategy has lower coverage and lower selective accuracy, yet better BAS-derived utility: it incurs fewer penalties overall. The key is that both answering strategies have accuracy below the break-even threshold of . Answering fewer questions can therefore improve utility even when the remaining answers are less accurate.
Thus, TCD in conjunction with the BAS metric is not guaranteed to improve either accuracy or coverage. If a practitioner values these metrics, the correct approach is to define a suitable downstream loss that penalizes for example low accuracy or coverage, and then run TCD with respect to this metric. If you are aware of a specific suitable loss function, we are happy to provide this experiment as well.
Rebuttal b3pr
Thank you for taking the time to review the paper. Overall we are glad that you see the relevance of our method and we think you raised some good points which we address below (new empirical results are present in Figshare)
W1: We really appreciate the suggestion to compare with a classifier that is trained on the task-specific loss, as we agree that this can make the paper significantly stronger. In particular, we train a linear classifier similar to TCD to minimize the empirical task loss. In specific, we train the following objective with L2 regularization:
\begin{equation} \min_{W,b} E_{(p,y)} \sum_{a \in \mathcal{A}} \operatorname{softmax}(Wp+b)_a \ell(a,y) \end{equation}
where p is the input predictive distribution, A is the action space, softmax(Wp+b)a denotes the probability assigned to action a, and ℓ(a,y) is the task-specific loss of taking action a when the ground-truth outcome is y. At inference time, we select the action with the highest predicted probability:
\begin{equation} \hat{a}(p) = \arg\max_{a \in \mathcal{A}} \operatorname{softmax}(Wp+b)_a. \end{equation}
While a strong baseline, we find it to underperform TCD, see Table 1 (Task-Classifier).
Furthermore, this baseline supports decision-making only for this fixed loss, whereas TCD reuses the same calibrated distribution across different losses without retraining. Our new ablation (Figure 3) explicitly visualizes this: across different thresholds t in the BAS Metric (each defining its own unique loss), TCD maintains strong performance while the baselines deteriorate. This is again distinct from the newly added task-dependent classifiers, which are tailored to a single loss (t=0.25) and are also shown to deteriorate in Figure 3.
W2: We agree that an exploration of different efficiency-performance trade-offs for both TCD and the baselines is insightful. We first present, in Table 1, ICL using 20 samples and MBR as the decision criterion, and find that TCD continues to outperform at this compute budget which matches TCD. For a more detailed efficiency-performance analysis, we present Figure 1, which shows latency versus task loss. In particular, we again evaluate ICL with additional samples to allow larger inference budgets. In our evaluation, TCD Pareto-dominates ICL, also at relatively low sampling budgets. Therefore, TCD enables a practitioner to trade off inference latency for task performance by adjusting the number of samples. These measurements include both LLM sampling and auxiliary semantic mapping (decomposition in Figure 2). The decomposition also shows that auxiliary mapping costs are rather low and the post-hoc decision procedure is negligible (microseconds).
W3: Our MMLU evaluation indeed only uses moral_scenarios, but we believe that our broad range of tasks and datasets sufficiently shows the practical efficacy of TCD. Should you think that another MMLU subset would strengthen the evaluation, we can also run experiments on this.
The discussion regarding distribution shifts is highly relevant: Like any other fine-tuning or post-hoc method, we rely on the labeled data to represent the data distribution at inference. Consequently, we do not expect the calibrator of TCD to be robust to strong distribution shifts. This is an important limitation that we will acknowledge explicitly in our revision (see updated manuscript). Addressing it, however, is beyond the scope of this work.
W4: Indeed, for weaker models, the performance gains in answer-abstain come from always abstaining. This behavior is consistent with the decision objective: for models with low answer accuracy, abstaining minimizes the expected incurred loss. Here, always abstaining may be the optimal policy, and a loss-aware decision rule should be able to recover this behavior. For stronger models, TCD allows to answer if the model confidence is sufficiently high, indicating that we do not abstain in general.
We also appreciate the suggestion to study different thresholds t for this answer-abstain problem. As mentioned above (W1), Figure 3 shows that TCD is the only method that maintains performance across different thresholds, whereas the baselines deteriorate. We also provide an accuracy-coverage plot across the 5 CV Folds for TriviaQA, for both calibrated and uncalibrated probabilities in Figure 4, which we find to differ only in their specific thresholds. However, we see this as somewhat orthogonal to TCD, as TCD, in this case, is optimized with respect to a decision under the BAS metric (specified for specific t) and produces not a score that can be thresholded, but instead directly the decision of whether to answer or abstain.
Thank you for your suggestions. We hope that the additional experiments help to strengthen the justification and performance of TCD. If there are further points, we are happy to discuss!