- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BAYESER, a Bayesian entropy-based decoding framework for masked diffusion language models. It constructs a checkpoint-centered posterior without fine-tuning the mean model, and uses posterior predictive entropy and confidence to determine token ordering, token selection, and commitment budgets during decoding. The main strength is the broad and detailed evaluation of Bayesian calibration for DLM decoding, while the main weakness is that the calibration gains come with non-negligible inference overhead and sometimes reduced generation quality compared with the simpler Base-Entropy baseline.
Critical strengths
(1) BAYESER can effectively calibrate the predictions of DLMs. The theoretical derivation in the paper is very detailed, and experiments are conducted on multiple datasets covering mathematics, code, and planning, as well as on two mainstream DLMs, demonstrating sufficient robustness of the method.
Critical weaknesses
(1) The method sacrifices inference quality and inference speed. Although calibration is improved, as shown in Tables 1 and 2, BAYESER shows quality degradation compared with the Base-Entropy method on four datasets. In addition, as discussed in Section 5.4, BAYESER increases inference latency, which makes the calibration gains appear to come at too high a cost.
Questions for authors
(1) BAYESER uses multiple posterior-sampled forward passes at each step. Does its improvement really come from the Bayesian posterior, or simply from using more inference compute / ensemble averaging?
4 comments
Reducing the cost with fewer posterior samples
We have added these results to Appendix C.2, "Posterior sample count: performance and efficiency" (Table 7) of the revised manuscript.
To examine how fewer posterior samples affect performance and cost, we vary S while fixing each dataset's posterior and decoding settings.
| Dataset | Samples S | Accuracy (%) ↑ | ECE ↓ | Brier ↓ | Generation min ↓ | Evaluation min ↓ | Peak allocated GiB ↓ |
|---|---|---|---|---|---|---|---|
| MBPP | 1 | 48.60 | .2824 | .3207 | 24.72 | 28.05 | 14.84 |
| MBPP | 2 | 51.80 | .2271 | .2872 | 27.36 | 30.72 | 15.29 |
| MBPP | 4 | 57.20 | .1650 | .2598 | 28.29 | 31.61 | 16.26 |
| MBPP | 8 | 55.80 | .1712 | .2619 | 32.20 | 35.68 | 18.15 |
| MBPP | 10 | 57.00 | .1546 | .2521 | 33.64 | 38.94 | 19.13 |
| HumanEval | 1 | 45.12 | .3395 | .3533 | 11.63 | 13.36 | 14.47 |
| HumanEval | 2 | 45.73 | .3196 | .3435 | 14.07 | 15.52 | 14.55 |
| HumanEval | 4 | 56.10 | .2002 | .2702 | 15.66 | 17.11 | 14.84 |
| HumanEval | 8 | 56.71 | .2021 | .2716 | 16.98 | 18.43 | 15.36 |
| HumanEval | 10 | 56.10 | .1812 | .2796 | 18.37 | 23.64 | 15.62 |
| GSM8K | 1 | 77.41 | .0800 | .1687 | 76.55 | 80.08 | 14.56 |
| GSM8K | 2 | 77.86 | .0710 | .1643 | 81.25 | 84.65 | 14.56 |
| GSM8K | 4 | 77.26 | .0814 | .1700 | 91.33 | 94.89 | 14.56 |
| GSM8K | 8 | 77.71 | .0707 | .1648 | 112.62 | 116.02 | 14.79 |
| GSM8K | 10 | 77.18 | .0771 | .1690 | 126.69 | 127.18 | 14.91 |
| MATH500 | 1 | 38.20 | .4796 | .4704 | 90.35 | 93.74 | 14.47 |
| MATH500 | 2 | 40.80 | .4516 | .4528 | 97.16 | 100.47 | 14.60 |
| MATH500 | 4 | 43.00 | .4303 | .4320 | 113.28 | 114.18 | 14.92 |
| MATH500 | 8 | 42.00 | .4391 | .4397 | 147.51 | 148.51 | 15.60 |
| MATH500 | 10 | 44.20 | .4187 | .4250 | 178.36 | 179.02 | 15.95 |
| Sudoku | 1 | 97.40 | .0283 | .0241 | 1.78 | 2.81 | 14.28 |
| Sudoku | 2 | 97.40 | .0357 | .0242 | 1.91 | 2.91 | 14.34 |
| Sudoku | 4 | 97.40 | .0329 | .0242 | 2.15 | 2.92 | 14.47 |
| Sudoku | 8 | 97.60 | .0338 | .0227 | 2.46 | 3.06 | 14.72 |
| Sudoku | 10 | 97.20 | .0342 | .0260 | 2.70 | 3.05 | 14.84 |
| Countdown | 1 | 25.50 | .3983 | .3232 | 37.85 | 38.63 | 14.31 |
| Countdown | 2 | 26.80 | .3586 | .2984 | 48.52 | 49.30 | 14.36 |
| Countdown | 4 | 27.60 | .3447 | .2865 | 59.06 | 59.82 | 14.48 |
| Countdown | 8 | 28.70 | .3250 | .2828 | 77.13 | 77.91 | 14.72 |
| Countdown | 10 | 30.90 | .3027 | .2726 | 89.26 | 90.26 | 14.84 |
Dropping S from 10 to 4 gives approximately the same accuracy and calibration on four of the six datasets (MBPP, HumanEval, GSM8K, and Sudoku), while cutting generation time by 14.8–27.9% and peak allocated memory by up to 15%. Specifically, reducing S from 10 to 4 lowers generation time by 15.9% on MBPP (57.20% vs 57.00% accuracy), 14.8% on HumanEval (56.10% at both), 27.9% on GSM8K (77.26% vs 77.18%), and 20.5% on Sudoku (97.40% vs 97.20%), with ECE and Brier close to their S=10 values. GSM8K remains stable even with one or two samples, and Sudoku remains stable with one.
On the other datasets, fewer samples trade performance for speed. On MATH500, four samples cut generation time by 36.5% at a modest accuracy cost (43.00% vs 44.20%), with similar ECE and Brier. Countdown benefits more from additional samples: four samples cut generation time by 33.8%, but accuracy drops from 30.90% to 27.60% and ECE rises from .3027 to .3447.
The sample count therefore lets users choose their own trade-off between cost and performance.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new ablation studies and computational efficiency analysis. While these results provide useful clarifications, I believe some concerns regarding the practical trade-offs and the specific benefits of Bayesian posterior estimation remain.
Overall, I have decided to maintain my current score at this stage. I would be happy to further discuss these concerns with the AC and other reviewers during the subsequent discussion.
Thank you again for your efforts and thoughtful responses.
We thank the reviewer for the response. We feel an additional experimental result would be beneficial to more explicitly address the reviewer's remaining concern of the computational cost vs accuracy-calibration trade-offs between BAYESER and simple averaging.
The prior-only A+B column of our covariance ablation (Section 5.5 and Table 4 of the revised paper) serves as the simple-averaging baseline. It uses the same sampled-decoding framework as BAYESER: S=10 weight samples, averaged predictive probabilities, entropy ordering and the adaptive commitment rule. It draws both LoRA factors from the isotropic prior, and , with no fitted curvature.
Table R1. Covariance and factor-sampling ablation (Table 4 of the revised paper)
Accuracy (%) ↑ / generated-content ECE ↓ / generated-content Brier ↓, with ECE and Brier multiplied by 100.
| Dataset | Full KFAC, A+B (BAYESER) | KFAC, fixed A | Prior-only, A+B (simple averaging) | Prior-only, fixed A |
|---|---|---|---|---|
| MBPP | 57.53 / 23.54 / 28.40 | 56.40 / 28.40 / 31.18 | 56.73 / 24.07 / 28.77 | 56.47 / 28.37 / 31.16 |
| HumanEval | 56.30 / 24.77 / 30.26 | 56.10 / 27.80 / 32.17 | 57.32 / 23.63 / 29.32 | 55.08 / 28.48 / 32.66 |
| GSM8K | 77.98 / 7.17 / 16.53 | 77.69 / 7.59 / 16.66 | 77.58 / 7.27 / 16.54 | 77.63 / 7.59 / 16.66 |
| MATH500 | 43.53 / 42.71 / 43.19 | 41.67 / 44.75 / 44.60 | 41.67 / 44.65 / 44.71 | 40.87 / 45.44 / 45.21 |
| Sudoku | 97.53 / 3.96 / 2.40 | 97.33 / 3.17 / 2.47 | 97.60 / 4.15 / 2.29 | 97.47 / 3.00 / 2.38 |
| Countdown | 32.13 / 28.98 / 26.71 | 23.97 / 58.88 / 51.86 | 31.23 / 29.29 / 26.13 | 23.40 / 59.37 / 52.18 |
Table R2. Recorded generation time and peak memory
Generation time for the full test set (min) on A100 / peak allocated GPU memory (GiB).
| Dataset | Full KFAC, A+B (BAYESER) | KFAC, fixed A | Prior-only, A+B (simple averaging) | Prior-only, fixed A | Base Entropy (no posterior samples) |
|---|---|---|---|---|---|
| MBPP | 26.84 / 19.13 | 26.44 / 19.13 | 27.51 / 19.12 | 26.42 / 19.13 | 18.56 / 14.76 |
| HumanEval | 16.13 / 15.56 | 14.46 / 15.55 | 16.10 / 15.62 | 14.41 / 15.56 | 11.63 / 14.38 |
| GSM8K | 123.55 / 14.91 | 123.30 / 14.89 | 119.60 / 14.89 | 119.77 / 14.85 | 58.85 / 14.56 |
| MATH500 | 132.39 / 15.92 | 116.83 / 15.93 | 115.28 / 15.93 | 114.05 / 15.93 | 71.77 / 14.38 |
| Sudoku | 2.94 / 14.84 | 2.42 / 14.84 | 2.67 / 14.84 | 2.39 / 14.84 | 1.37 / 14.26 |
| Countdown | 77.08 / 14.84 | 45.08 / 14.84 | 76.64 / 14.84 | 43.44 / 14.84 | 23.95 / 14.27 |
Relative to the simple averaging with prior-only A+B, BAYESER improves accuracy, ECE and Brier on MBPP, GSM8K and MATH500. The gain is largest on MATH500, with an accuracy gain of 1.86 percentage points, while ECE decreases from 44.65 to 42.71 and Brier from 44.71 to 43.19; on MBPP and GSM8K, accuracy rises by 0.80 and 0.40 points. BAYESER also has lower ECE on Sudoku and Countdown and higher accuracy on Countdown, while HumanEval favors simple averaging. These gains come at almost no extra decoding cost: with the same number of samples, the peak allocated memory is the same, and the generation time is within −2.4% to +3.3% of simple averaging on MBPP, HumanEval, GSM8K, and Countdown, +14.8% on MATH500, and 0.27 minutes longer on Sudoku.
Compared with Base Entropy, which draws no posterior samples, BAYESER takes 39–115% more generation time and 2–30% more peak memory on MBPP, HumanEval, GSM8K, MATH500 and Sudoku. In return it raises accuracy by 3.53 points on MATH500 and lowers generated-content ECE on MBPP, HumanEval and MATH500. On Countdown, BAYESER takes 3.2× the generation time of Base Entropy with 4% more memory, mainly because BAYESER's adaptive budget commits about one token per step. This more cautious commitment comes with the largest gain: accuracy rises by 8.23 points (23.90% → 32.13%) and ECE is roughly halved (28.98 vs 60.07).
BAYESER therefore trades computation for accuracy and calibration. This cost can be further lowered by drawing fewer posterior samples: in our sample-count study (Appendix C.2 and Table 7 of the revised paper), reducing S from 10 to 4 cuts generation time by 15.9%, 14.8%, 27.9%, 36.5% and 20.5% on MBPP, HumanEval, GSM8K, MATH500, and Sudoku while keeping accuracy and calibration comparable.
Thank you for emphasizing the computational overheads needed to achieve the accuracy and calibration gains. We agree that BAYESER involves a trade-off: it spends additional inference computation to obtain better-calibrated confidence and, on some tasks, higher accuracy. Below we (1) quantify this additional cost, (2) show that it comes from posterior sampling and averaging, and that this sampling, not just the extra compute, contributes to the gains, and (3) show that much of the cost can be removed by drawing fewer posterior samples. The new controls show that joint predictive sampling contributes substantially to the observed improvements, while fitted curvature provides additional, task-dependent benefits.
Computational overhead
We agree that the overhead is meaningful. Section 5.4, Computational efficiency, reports measured latency and memory with ten posterior samples. On HumanEval/A100, one uncertain layer takes 17.4–22.1 minutes versus 13.4 for Thresholding (1.30–1.65×). Across the tested depths, peak allocated memory is 15.54–15.71 GiB versus 14.38 GiB, an increase of about 8–9%. Activations before the first uncertain layer are shared across samples; only the remaining layers are re-evaluated per draw. This keeps the additional cost limited for shallow posteriors, while deeper configurations incur higher latency. With one uncertain layer, BAYESER therefore trades 1.30–1.65× the decoding time and about 8–9% more memory for its calibration gains. The next two sections show what this cost buys and how much of it can be removed.
Do the improvements really come from the Bayesian posterior?
The computational overhead mainly comes from evaluating the forward pass for each of the S posterior samples of A and B and averaging the resulting predictive distributions. Using two additional ablations, reported in full in our response to Reviewer ewH1, we show that this sampling and averaging contributes to the gains, and that the gains are not explained by the extra forward passes alone: within each ablation, all variants use the same number of posterior samples (S=10), so they have the same number of forward passes per step and differ only in how the samples are drawn or used.
In “Ablation: KFAC, prior-only covariance, and fixed-A sampling” (Section 5.5, Table 4), we compare fitted KFAC and prior-only covariance, either sampling both LoRA factors or fixing A, while keeping the number of draws the same. Joint A+B sampling gives the more consistent calibration benefit: it lowers ECE and Brier relative to fixing A on five of six datasets, and on Countdown, prior-only A+B raises accuracy from 23.40% to 31.23% while roughly halving ECE and Brier. The full KFAC reference further improves accuracy and calibration over prior-only A+B on MBPP, GSM8K, and MATH500; on MATH500, accuracy rises from 41.67% to 43.53% and ECE falls from .4465 to .4271. These findings support the value of fitted KFAC alongside joint factor sampling.
In “Ablation: ordering versus token proposals” (Appendix G.2, Table 14), we compare base ordering/base tokens, posterior ordering/base tokens, and posterior ordering/posterior tokens under a common commitment budget. At K=1, posterior ordering improves Countdown accuracy from 23.70% to 29.20%, and posterior token selection provides additional accuracy gains on four datasets and further lowers both ECE and Brier on MBPP, GSM8K, and MATH500. Because these variants change only the ordering and token proposals, the gains reflect changes in the generated answers, not just in the reported confidence. At K=2, posterior ordering does not show consistent improvements, showing that its benefit depends on the commitment budget. The schedule traces (Appendix G.1, Table 13) show that adaptive BAYESER commits one token in most of its early steps. Together, these results connect the K=1 gains to a commitment size frequently used by the full method.
Overall, at the same sampling cost, the choice of how posterior samples are drawn and used changes both accuracy and calibration. The additional computation therefore buys the effect of the posterior predictive. Joint predictive averaging accounts for a substantial part of the improvement, and fitted curvature adds a further, task-dependent benefit.