acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Entropy-based Reordering for Calibrated Diffusion Language Models

Abstract

Masked Diffusion Language Models (MDLMs) generate sequences by iteratively replacing masked tokens with model predictions. At each denoising step, the decoder chooses which positions are sufficiently confident to commit. Existing decoding methods typically rely on softmax confidence, which can be miscalibrated. We introduce *BayesER* (*BAYES*ian *E*ntropy-based *R*eordering), a post-hoc Bayesian decoding framework that uses predictive uncertainty to guide token commitment. In BayesER, we construct a lightweight approximate posterior centered at the pretrained checkpoint, similar to Laplace-LoRA but without training LoRA adapters. We average predictions over posterior samples and use predictive entropy to prioritize reliable positions. We examine how posterior predictions affect position ordering and token selection across benchmarks spanning code generation, mathematical reasoning, planning, and molecular generation. We show that BayesER reduces sequence-level calibration error while preserving or improving accuracy relative to common decoding schemes, including confidence-threshold decoding. Additionally, a posterior fitted on one code-generation dataset reduces calibration error on another without refitting, suggesting that Bayesian uncertainty may provide a transferable signal for more reliable MDLM decoding.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Masked Diffusion Language Models, Bayesian Deep Learning, Uncertainty Estimation, Calibration, Language Model Decoding, Predictive Uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q14dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BAYESER, a Bayesian entropy-based decoding framework for masked diffusion language models. It constructs a checkpoint-centered posterior without fine-tuning the mean model, and uses posterior predictive entropy and confidence to determine token ordering, token selection, and commitment budgets during decoding. The main strength is the broad and detailed evaluation of Bayesian calibration for DLM decoding, while the main weakness is that the calibration gains come with non-negligible inference overhead and sometimes reduced generation quality compared with the simpler Base-Entropy baseline.

Critical strengths

(1) BAYESER can effectively calibrate the predictions of DLMs. The theoretical derivation in the paper is very detailed, and experiments are conducted on multiple datasets covering mathematics, code, and planning, as well as on two mainstream DLMs, demonstrating sufficient robustness of the method.

Critical weaknesses

(1) The method sacrifices inference quality and inference speed. Although calibration is improved, as shown in Tables 1 and 2, BAYESER shows quality degradation compared with the Base-Entropy method on four datasets. In addition, as discussed in Section 5.4, BAYESER increases inference latency, which makes the calibration gains appear to come at too high a cost.

Questions for authors

(1) BAYESER uses multiple posterior-sampled forward passes at each step. Does its improvement really come from the Bayesian posterior, or simply from using more inference compute / ensemble averaging?

4 comments

Thank you for emphasizing the computational overheads needed to achieve the accuracy and calibration gains. We agree that BAYESER involves a trade-off: it spends additional inference computation to obtain better-calibrated confidence and, on some tasks, higher accuracy. Below we (1) quantify this additional cost, (2) show that it comes from posterior sampling and averaging, and that this sampling, not just the extra compute, contributes to the gains, and (3) show that much of the cost can be removed by drawing fewer posterior samples. The new controls show that joint predictive sampling contributes substantially to the observed improvements, while fitted curvature provides additional, task-dependent benefits.

Computational overhead

We agree that the overhead is meaningful. Section 5.4, Computational efficiency, reports measured latency and memory with ten posterior samples. On HumanEval/A100, one uncertain layer takes 17.4–22.1 minutes versus 13.4 for Thresholding (1.30–1.65×). Across the tested depths, peak allocated memory is 15.54–15.71 GiB versus 14.38 GiB, an increase of about 8–9%. Activations before the first uncertain layer are shared across samples; only the remaining layers are re-evaluated per draw. This keeps the additional cost limited for shallow posteriors, while deeper configurations incur higher latency. With one uncertain layer, BAYESER therefore trades 1.30–1.65× the decoding time and about 8–9% more memory for its calibration gains. The next two sections show what this cost buys and how much of it can be removed.

Do the improvements really come from the Bayesian posterior?

The computational overhead mainly comes from evaluating the forward pass for each of the S posterior samples of A and B and averaging the resulting predictive distributions. Using two additional ablations, reported in full in our response to Reviewer ewH1, we show that this sampling and averaging contributes to the gains, and that the gains are not explained by the extra forward passes alone: within each ablation, all variants use the same number of posterior samples (S=10), so they have the same number of forward passes per step and differ only in how the samples are drawn or used.

In “Ablation: KFAC, prior-only covariance, and fixed-A sampling” (Section 5.5, Table 4), we compare fitted KFAC and prior-only covariance, either sampling both LoRA factors or fixing A, while keeping the number of draws the same. Joint A+B sampling gives the more consistent calibration benefit: it lowers ECE and Brier relative to fixing A on five of six datasets, and on Countdown, prior-only A+B raises accuracy from 23.40% to 31.23% while roughly halving ECE and Brier. The full KFAC reference further improves accuracy and calibration over prior-only A+B on MBPP, GSM8K, and MATH500; on MATH500, accuracy rises from 41.67% to 43.53% and ECE falls from .4465 to .4271. These findings support the value of fitted KFAC alongside joint factor sampling.

In “Ablation: ordering versus token proposals” (Appendix G.2, Table 14), we compare base ordering/base tokens, posterior ordering/base tokens, and posterior ordering/posterior tokens under a common commitment budget. At K=1, posterior ordering improves Countdown accuracy from 23.70% to 29.20%, and posterior token selection provides additional accuracy gains on four datasets and further lowers both ECE and Brier on MBPP, GSM8K, and MATH500. Because these variants change only the ordering and token proposals, the gains reflect changes in the generated answers, not just in the reported confidence. At K=2, posterior ordering does not show consistent improvements, showing that its benefit depends on the commitment budget. The schedule traces (Appendix G.1, Table 13) show that adaptive BAYESER commits one token in most of its early steps. Together, these results connect the K=1 gains to a commitment size frequently used by the full method.

Overall, at the same sampling cost, the choice of how posterior samples are drawn and used changes both accuracy and calibration. The additional computation therefore buys the effect of the posterior predictive. Joint predictive averaging accounts for a substantial part of the improvement, and fitted curvature adds a further, task-dependent benefit.

Reducing the cost with fewer posterior samples

We have added these results to Appendix C.2, "Posterior sample count: performance and efficiency" (Table 7) of the revised manuscript.

To examine how fewer posterior samples affect performance and cost, we vary S while fixing each dataset's posterior and decoding settings.

DatasetSamples SAccuracy (%) ↑ECE ↓Brier ↓Generation min ↓Evaluation min ↓Peak allocated GiB ↓
MBPP148.60.2824.320724.7228.0514.84
MBPP251.80.2271.287227.3630.7215.29
MBPP457.20.1650.259828.2931.6116.26
MBPP855.80.1712.261932.2035.6818.15
MBPP1057.00.1546.252133.6438.9419.13
HumanEval145.12.3395.353311.6313.3614.47
HumanEval245.73.3196.343514.0715.5214.55
HumanEval456.10.2002.270215.6617.1114.84
HumanEval856.71.2021.271616.9818.4315.36
HumanEval1056.10.1812.279618.3723.6415.62
GSM8K177.41.0800.168776.5580.0814.56
GSM8K277.86.0710.164381.2584.6514.56
GSM8K477.26.0814.170091.3394.8914.56
GSM8K877.71.0707.1648112.62116.0214.79
GSM8K1077.18.0771.1690126.69127.1814.91
MATH500138.20.4796.470490.3593.7414.47
MATH500240.80.4516.452897.16100.4714.60
MATH500443.00.4303.4320113.28114.1814.92
MATH500842.00.4391.4397147.51148.5115.60
MATH5001044.20.4187.4250178.36179.0215.95
Sudoku197.40.0283.02411.782.8114.28
Sudoku297.40.0357.02421.912.9114.34
Sudoku497.40.0329.02422.152.9214.47
Sudoku897.60.0338.02272.463.0614.72
Sudoku1097.20.0342.02602.703.0514.84
Countdown125.50.3983.323237.8538.6314.31
Countdown226.80.3586.298448.5249.3014.36
Countdown427.60.3447.286559.0659.8214.48
Countdown828.70.3250.282877.1377.9114.72
Countdown1030.90.3027.272689.2690.2614.84

Dropping S from 10 to 4 gives approximately the same accuracy and calibration on four of the six datasets (MBPP, HumanEval, GSM8K, and Sudoku), while cutting generation time by 14.8–27.9% and peak allocated memory by up to 15%. Specifically, reducing S from 10 to 4 lowers generation time by 15.9% on MBPP (57.20% vs 57.00% accuracy), 14.8% on HumanEval (56.10% at both), 27.9% on GSM8K (77.26% vs 77.18%), and 20.5% on Sudoku (97.40% vs 97.20%), with ECE and Brier close to their S=10 values. GSM8K remains stable even with one or two samples, and Sudoku remains stable with one.

On the other datasets, fewer samples trade performance for speed. On MATH500, four samples cut generation time by 36.5% at a modest accuracy cost (43.00% vs 44.20%), with similar ECE and Brier. Countdown benefits more from additional samples: four samples cut generation time by 33.8%, but accuracy drops from 30.90% to 27.60% and ECE rises from .3027 to .3447.

The sample count therefore lets users choose their own trade-off between cost and performance.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new ablation studies and computational efficiency analysis. While these results provide useful clarifications, I believe some concerns regarding the practical trade-offs and the specific benefits of Bayesian posterior estimation remain.

Overall, I have decided to maintain my current score at this stage. I would be happy to further discuss these concerns with the AC and other reviewers during the subsequent discussion.

Thank you again for your efforts and thoughtful responses.

We thank the reviewer for the response. We feel an additional experimental result would be beneficial to more explicitly address the reviewer's remaining concern of the computational cost vs accuracy-calibration trade-offs between BAYESER and simple averaging.

The prior-only A+B column of our covariance ablation (Section 5.5 and Table 4 of the revised paper) serves as the simple-averaging baseline. It uses the same sampled-decoding framework as BAYESER: S=10 weight samples, averaged predictive probabilities, entropy ordering and the adaptive commitment rule. It draws both LoRA factors from the isotropic prior, A∼N(A0,λ−1I)A \sim \mathcal{N}(A_0, \lambda^{-1} I) and B∼N(0,λ−1I)B \sim \mathcal{N}(0, \lambda^{-1} I), with no fitted curvature.

Table R1. Covariance and factor-sampling ablation (Table 4 of the revised paper)

Accuracy (%) ↑ / generated-content ECE ↓ / generated-content Brier ↓, with ECE and Brier multiplied by 100.

DatasetFull KFAC, A+B (BAYESER)KFAC, fixed APrior-only, A+B (simple averaging)Prior-only, fixed A
MBPP57.53 / 23.54 / 28.4056.40 / 28.40 / 31.1856.73 / 24.07 / 28.7756.47 / 28.37 / 31.16
HumanEval56.30 / 24.77 / 30.2656.10 / 27.80 / 32.1757.32 / 23.63 / 29.3255.08 / 28.48 / 32.66
GSM8K77.98 / 7.17 / 16.5377.69 / 7.59 / 16.6677.58 / 7.27 / 16.5477.63 / 7.59 / 16.66
MATH50043.53 / 42.71 / 43.1941.67 / 44.75 / 44.6041.67 / 44.65 / 44.7140.87 / 45.44 / 45.21
Sudoku97.53 / 3.96 / 2.4097.33 / 3.17 / 2.4797.60 / 4.15 / 2.2997.47 / 3.00 / 2.38
Countdown32.13 / 28.98 / 26.7123.97 / 58.88 / 51.8631.23 / 29.29 / 26.1323.40 / 59.37 / 52.18

Table R2. Recorded generation time and peak memory

Generation time for the full test set (min) on A100 / peak allocated GPU memory (GiB).

DatasetFull KFAC, A+B (BAYESER)KFAC, fixed APrior-only, A+B (simple averaging)Prior-only, fixed ABase Entropy (no posterior samples)
MBPP26.84 / 19.1326.44 / 19.1327.51 / 19.1226.42 / 19.1318.56 / 14.76
HumanEval16.13 / 15.5614.46 / 15.5516.10 / 15.6214.41 / 15.5611.63 / 14.38
GSM8K123.55 / 14.91123.30 / 14.89119.60 / 14.89119.77 / 14.8558.85 / 14.56
MATH500132.39 / 15.92116.83 / 15.93115.28 / 15.93114.05 / 15.9371.77 / 14.38
Sudoku2.94 / 14.842.42 / 14.842.67 / 14.842.39 / 14.841.37 / 14.26
Countdown77.08 / 14.8445.08 / 14.8476.64 / 14.8443.44 / 14.8423.95 / 14.27

Relative to the simple averaging with prior-only A+B, BAYESER improves accuracy, ECE and Brier on MBPP, GSM8K and MATH500. The gain is largest on MATH500, with an accuracy gain of 1.86 percentage points, while ECE decreases from 44.65 to 42.71 and Brier from 44.71 to 43.19; on MBPP and GSM8K, accuracy rises by 0.80 and 0.40 points. BAYESER also has lower ECE on Sudoku and Countdown and higher accuracy on Countdown, while HumanEval favors simple averaging. These gains come at almost no extra decoding cost: with the same number of samples, the peak allocated memory is the same, and the generation time is within −2.4% to +3.3% of simple averaging on MBPP, HumanEval, GSM8K, and Countdown, +14.8% on MATH500, and 0.27 minutes longer on Sudoku.

Compared with Base Entropy, which draws no posterior samples, BAYESER takes 39–115% more generation time and 2–30% more peak memory on MBPP, HumanEval, GSM8K, MATH500 and Sudoku. In return it raises accuracy by 3.53 points on MATH500 and lowers generated-content ECE on MBPP, HumanEval and MATH500. On Countdown, BAYESER takes 3.2× the generation time of Base Entropy with 4% more memory, mainly because BAYESER's adaptive budget commits about one token per step. This more cautious commitment comes with the largest gain: accuracy rises by 8.23 points (23.90% → 32.13%) and ECE is roughly halved (28.98 vs 60.07).

BAYESER therefore trades computation for accuracy and calibration. This cost can be further lowered by drawing fewer posterior samples: in our sample-count study (Appendix C.2 and Table 7 of the revised paper), reducing S from 10 to 4 cuts generation time by 15.9%, 14.8%, 27.9%, 36.5% and 20.5% on MBPP, HumanEval, GSM8K, MATH500, and Sudoku while keeping accuracy and calibration comparable.

Reviewer_rtJyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper provides a Bayesian framework for Entropy-based Ordering in calibrated diffusion language models. The key idea is to compute a lightweight approximate posterior centered at the pretrained checkpoint.

Critical strengths

New framework to select which tokens to unmark.

Well-executed paper

Critical weaknesses

Could be written much more clearly

Not clear whether the results are ground-breaking

Questions for authors

N/A

1 comment

Thank you for recognizing the token-commitment framework and the execution of the study. We would like to clarify three contributions:

  • A checkpoint-centered approximate posterior that provides predictive uncertainty without fine-tuning the model mean.
  • A decoding framework that uses this uncertainty to guide position ordering, token selection, and adaptive commitment counts.
  • An evaluation of accuracy, calibration, and computational cost across code, mathematics, and planning, with additional ablations examining the contributions of fitted curvature and posterior-based decoding decisions.

The significance is in using Bayesian uncertainty to guide the generation process as well as estimate confidence in its outputs. We have revised the writing to make the motivation and decoding procedure clearer (changes in blue). We would appreciate specific suggestions on which sections or explanations were hardest to follow.

Reviewer_ewH1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BAYESER addresses overconfident token commitment in masked diffusion language models through approximate Bayesian prediction. It fits a checkpoint-centered Gaussian over zero-update LoRA adapters using KFAC model-Fisher curvature, without fine-tuning the adapter means. During decoding, it averages sampled-model token distributions, prioritizes positions with low predictive entropy, selects tokens from the predictive mean, and sets the number of commitments using posterior confidence. It evaluates sequence confidence by aggregating probabilities recorded when tokens were committed.

The main comparisons are against threshold decoding (Fast-dLLM), Base Entropy, and Info-gain on Dream-7B, with additional LLaDA and GenMol experiments. Results show task-dependent accuracy-calibration trade-offs. For example, the calibration-oriented setting reduces Dream/MBPP ECE from 0.2918 to 0.1632 while accuracy changes from 57.60% to 56.73%. Transferring the MBPP posterior and settings to HumanEval also improves calibration without refitting. Sensitivity analyses, mutual-information ordering comparisons, and runtime measurements examine the method's operating conditions and costs.

Critical strengths
  1. The calibration improvements and MBPP-to-HumanEval transfer provide useful empirical evidence without task-specific fine-tuning. On Dream/MBPP, the calibration-oriented setting reduces generated-content ECE from 0.2918 to 0.1632 and Brier score from 0.3185 to 0.2557, with accuracy changing from 57.60% to 56.73% (Table 1). Reusing the MBPP posterior and settings on HumanEval reduces ECE from 0.3090 to 0.2016 and Brier score from 0.3327 to 0.2776, with 55.08% versus 54.88% accuracy. The latter is a meaningful transfer result because curvature is not refitted on the target benchmark.

  2. The evaluation documents a broad range of outcomes and computational trade-offs. It covers Dream, LLaDA, and molecular generation with GenMol, rather than only one model family or task. Dream/Countdown improves from threshold decoding's 18.90% accuracy to 32.13%, with lower ECE and Brier score, although Info-gain achieves higher accuracy at 41.47% (Table 2). The paper also reports unfavorable cases, including LLaDA/MATH500, where accuracy falls from 38.00% to 35.40% and calibration worsens (Appendix D). Posterior-depth/prior sweeps, MI-ordering comparisons, and measured latency and memory make important limitations visible.

Critical weaknesses
  1. The independence of configuration selection and final evaluation is unclear. Tables 1-2 report separate "best accuracy" and "best calibration" settings, and Table 5 specifies task-dependent posterior depths, prior precisions, and commitment thresholds. However, Section 5 and Appendix A do not identify the independent validation sets or selection rules used to choose these settings. Curvature-fitting sources are described at a high level, but exact dataset/subset identities, sizes, and masking procedures are not fully specified for the main tasks. The reproducibility statement explicitly excludes planning test inputs from synthetic training data; this addresses fitting overlap, but does not resolve hyperparameter-selection independence. Consequently, the reported gains cannot yet be distinguished from possible selection effects. This is an unresolved protocol issue, not evidence of test contamination. The MBPP-to-HumanEval transfer is valuable but does not resolve selection for the other benchmarks.

  2. The added value of Bayesian curvature, relative to simpler alternatives and its computational cost, is not established. BAYESER jointly changes token predictions, position ordering, commitment counts, and confidence scores. Base Entropy and MI ordering provide useful controls, but do not isolate data-informed curvature from generic predictive smoothing or a change in confidence scale. A simple held-out recalibration of sequence scores on fixed outputs and a perturbation control without fitted curvature are missing. This matters because Equation (7) is a candidate task-confidence score, and the construction is explicitly checkpoint-centered rather than fitted at a task-posterior mode. Moreover, Section 3.1 acknowledges that the A-factor Fisher block is zero at initialization: A variation is prior-controlled, and nonlinear sampling introduces a bilinear delta-B times delta-A term. These are acknowledged approximation choices, not proof that the decoder is invalid; their empirical contribution needs testing.

  3. The paper neither discusses nor compares against some related works, for instance, CCD (Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models, ICML 2026). CCD aggregates historical predictions to guide token commitment without additional model forward passes and is evaluated on both Dream and LLaDA across overlapping mathematical reasoning and code-generation benchmarks. It is therefore a relevant baseline for assessing the benefit of BAYESER’s posterior sampling. A direct comparison with CCD and its adaptive variant, CCD-DS, under a common evaluation protocol should report task accuracy, calibration, and measured latency. This would clarify whether BAYESER’s posterior-based predictions offer benefits that justify the additional computational cost.

Questions for authors
  1. Please identify the curvature-fitting data and subsets, their sizes and masking procedure, and the data used to select each Table 5 configuration. What objective and search budget define "best accuracy" and "best calibration"? Were the final evaluation examples excluded from all outcome-based selection? Please provide independently selected evaluation results if that separation was not maintained.

  2. How does BAYESER compare with CCD and CCD-DS on representative overlapping tasks using the same checkpoint, prompts and generation limits? Please report accuracy, ECE/Brier under the same commitment-time confidence aggregation, actual latency, and memory.

  3. How much of the calibration gain remains against a simple held-out recalibration of the threshold/Base Entropy sequence-confidence scores with their generated outputs fixed? Separately, can a focused ablation change posterior-based ordering while retaining base token proposals and a common commitment schedule? These comparisons would distinguish better task-confidence estimates from improved commitment decisions; they need not assume that recalibration improves task accuracy.

7 comments

Thank you for the detailed and constructive review. We agree that the manuscript should explain the fitting data and configuration selection more clearly. Here we give those which are added to the revised paper.

Curvature-fitting data and independence of configuration selection

Fitting data. We estimate curvature using training examples: 374 from MBPP, 6,973 from GSM8K, 7,000 from MATH, and 7,500 generated examples each for Sudoku and Countdown. HumanEval reuses the MBPP posterior without refitting.

Configuration selection. For the revised evaluation, we use MBPP's official 90-problem validation split and 500 validation problems for each of GSM8K, MATH, Sudoku, and Countdown. The GSM8K and MATH validation problems are held out from training before curvature fitting. Planning validation problems are generated separately, excluding training and test inputs.

We search uncertain-layer depths {1, 2, 4}, commitment thresholds {0.85, 0.90}, and prior-precision {200, 300, 500, 1000, 2000}. Accuracy-oriented settings use the highest observed validation accuracy; balanced settings are chosen using the validation trade-off between accuracy, generated-content ECE, and decoding cost. We then fix these configurations and evaluate them on the full test sets with different seeds. HumanEval inherits both the posterior and decoding settings selected on MBPP, without HumanEval-specific tuning.

Fixed-output recalibration

Following the reviewer's suggestion, we tested whether recalibrating each decoder's confidence after decoding gives a similar benefit. We have added this section to Appendix F of our revised manuscript. The outputs are unchanged, so accuracy is unchanged. For Thresholding, Base Entropy and BAYESER, we recalibrate each confidence ss as σ(a⋅logit(s)+b)\sigma(a \cdot \mathrm{logit}(s) + b) with a≥0a \ge 0. The map is monotone nondecreasing; it preserves rankings when a>0a>0 and becomes a constant predictor when a=0a=0.

We fit aa and bb on held-out validation problems disjoint from the test sets and apply the map unchanged to the test set. HumanEval has no validation split and reuses the MBPP maps. BAYESER's map is fitted on the validation set with its selected configuration and applied to three test seeds; its rows are seed means.

We report three versions of each score: Raw, Recalibrated, and Constant, which gives every output the decoder's validation accuracy. The constant uses the same labels but cannot tell correct outputs from incorrect ones. Cells report generated-content ECE ↓ / Brier ↓.

DatasetDecoderAccuracy (%) ↑RawRecalibratedConstant
MBPPThresholding57.60.2918 / .3185.0490 / .2295.0018 / .2442
MBPPBase Entropy57.00.2899 / .3123.0737 / .2246.0411 / .2468
MBPPBAYESER56.60.1598 / .2555.0551 / .2302.0229 / .2462
HumanEvalThresholding54.88.3090 / .3327.0802 / .2379.0290 / .2485
HumanEvalBase Entropy57.32.2725 / .3180.0832 / .2501.0379 / .2461
HumanEvalBAYESER55.69.1908 / .2776.0645 / .2440.0320 / .2478
GSM8KThresholding75.66.1029 / .1759.0469 / .1661.0474 / .1864
GSM8KBase Entropy77.63.0750 / .1656.0229 / .1604.0097 / .1737
GSM8KBAYESER77.61.0729 / .1651.0271 / .1607.0179 / .1741
MATH500Thresholding42.40.4550 / .4533.0394 / .2456.0360 / .2455
MATH500Base Entropy40.00.4627 / .4577.0105 / .2403.0020 / .2400
MATH500BAYESER42.40.4345 / .4354.0120 / .2444.0120 / .2444
SudokuThresholding97.80.0337 / .0212.0116 / .0215.0120 / .0217
SudokuBase Entropy97.60.0306 / .0225.0047 / .0214.0060 / .0235
SudokuBAYESER97.33.0341 / .0249.0130 / .0250.0147 / .0262
CountdownThresholding18.90.6646 / .5856.0650 / .1177.0330 / .1544
CountdownBase Entropy23.90.6007 / .5321.0577 / .1564.0470 / .1841
CountdownBAYESER31.97.2924 / .2693.0855 / .1659.0563 / .2207

Low ECE after recalibration is easy to obtain. Recalibration lowers ECE in all 18 rows and Brier in 16. Yet the constant, which cannot separate correct from incorrect outputs, has even lower ECE in 13 of 18 rows, while the recalibrated score wins on Brier in 14 of 18. Recalibrated ECE is therefore not comparable with the raw, label-free scores in Tables 1 and 2; the fair comparisons are raw against raw and recalibrated against recalibrated.

BAYESER benefits just as much. Because the decoders generate different outputs with different correctness rates, cross-decoder Brier differences do not isolate confidence-estimator quality. We therefore report accuracy and improvement relative to each decoder’s validation-accuracy constant alongside the absolute Brier scores.

The same map lowers BAYESER's ECE on every dataset, e.g. from .160 to .055 on MBPP and from .292 to .086 on Countdown. After recalibration the three decoders are close: BAYESER is within .03 ECE of the best baseline on every dataset and within .007 Brier outside Countdown.

Post-hoc recalibration requires labeled examples to fit a confidence-to-correctness mapping. In our transfer experiment, these examples come from MBPP rather than HumanEval, so both the recalibration maps and BAYESER’s posterior/settings transfer without HumanEval-specific fitting. BAYESER additionally uses its predictive distributions during generation, whereas the post-hoc map changes only the reported confidence.

Comparison with CCD and CCD-DS

We have added CCD and CCD-DS to Table 1 (mathematical reasoning and programming) and Table 2 (planning) of the revised manuscript. We use their released decoding policies, with K=1 and expansion ratio 4. Each cell below reports accuracy (%) ↑ / generated-content ECE ↓ / generated-content Brier ↓. Thresholding and Base Entropy retain their main-table results except for GSM8K and Sudoku, which use the fresh seed-0 baseline runs from the recalibration study above.

DatasetThresholdingBase EntropyCCDCCD-DSBAYESER
MBPP57.60 / .2918 / .318557.00 / .2899 / .312355.20 / .3031 / .323839.60 / .3367 / .324256.60 / .1598 / .2555
HumanEval54.88 / .3090 / .332757.32 / .2725 / .318052.44 / .3211 / .349032.32 / .3984 / .364355.69 / .1908 / .2776
GSM8K75.66 / .1029 / .175977.63 / .0750 / .165678.39 / .0780 / .161965.43 / .0996 / .215877.61 / .0732 / .1657
MATH50042.40 / .4550 / .453340.00 / .4627 / .457739.60 / .4755 / .470020.40 / .5637 / .491442.40 / .4345 / .4354
Sudoku97.80 / .0337 / .021297.60 / .0306 / .022597.60 / .0159 / .021797.60 / .0171 / .021697.33 / .0341 / .0249
Countdown18.90 / .6646 / .585623.90 / .6007 / .532118.10 / .6601 / .570813.20 / .6259 / .493631.97 / .2924 / .2693

BAYESER has higher accuracy and lower ECE/Brier than CCD on MBPP, HumanEval, and MATH500. CCD achieves higher GSM8K accuracy and lower Brier, while BAYESER has slightly lower ECE. On Sudoku, CCD and CCD-DS reach 97.60% accuracy versus 97.80% for Thresholding and 97.33% for BAYESER, and both have lower ECE and Brier than BAYESER.

The larger CCD-DS accuracy drop compared to the original paper may partly reflect differences from its original evaluation protocol: our evaluations use sliding windows of 16–128 tokens and shorter code-generation limits. Under this diffusion generation geometry, agreement across successive steps may be a less reliable signal for committing several tokens together.

Ablation: KFAC, prior-only covariance, and fixed-A sampling

We have added this ablation to Section 5.5, "Ablation: KFAC, prior-only covariance, and fixed-A sampling" (Table 4) of the revised manuscript.

To assess how fitted KFAC curvature and prior-driven perturbations contribute to accuracy and calibration, we compare four ways of sampling the LoRA factors A and B:

  • Full KFAC, A+B: sample both factors, using fitted KFAC curvature and the prior for B, while A variation is prior-controlled.
  • KFAC, fixed A: keep A fixed and sample only B using the fitted KFAC covariance.
  • Prior-only, A+B: remove fitted curvature and sample both factors using isotropic prior covariance.
  • Prior-only, fixed A: keep A fixed and sample only B using isotropic prior covariance.

We evaluate the following variants using S=10 weight samples. Cells report accuracy (%) ↑ / generated-content ECE ↓ / generated-content Brier ↓, averaged over three seeds. ECE and Brier are on the 0–1 scale.

DatasetFull KFAC, A+BKFAC, fixed APrior-only, A+BPrior-only, fixed A
MBPP57.53 / .2354 / .284056.40 / .2840 / .311856.73 / .2407 / .287756.47 / .2837 / .3116
HumanEval56.30 / .2477 / .302656.10 / .2780 / .321757.32 / .2363 / .293255.08 / .2848 / .3266
GSM8K77.98 / .0717 / .165377.69 / .0759 / .166677.58 / .0727 / .165477.63 / .0759 / .1666
MATH50043.53 / .4271 / .431941.67 / .4475 / .446041.67 / .4465 / .447140.87 / .4544 / .4521
Sudoku97.53 / .0396 / .024097.33 / .0317 / .024797.60 / .0415 / .022997.47 / .0300 / .0238
Countdown32.13 / .2898 / .267123.97 / .5888 / .518631.23 / .2929 / .261323.40 / .5937 / .5218

Sampling both A and B improves generated-content ECE and Brier over fixing A on five of the six datasets. Countdown provides the strongest new evidence: prior-only A+B improves accuracy from 23.40% to 31.23% and roughly halves ECE and Brier relative to prior-only fixed A. Sudoku is a trade-off: A+B slightly improves accuracy and Brier, while fixing A gives lower ECE.

Among the A+B variants, the full KFAC reference gives further mean improvements in accuracy and calibration on MBPP, GSM8K, and MATH500. HumanEval favors prior-only A+B; on Sudoku, KFAC has lower ECE but slightly worse accuracy and Brier than prior-only A+B. The results favor joint A+B sampling in most settings, with an additional, task-dependent advantage for the KFAC A+B reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.