- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether intermediate states along a single denoising trajectory of a masked large language diffusion model (LLDM) can replace the M independent generations that sample-based semantic UQ needs. It proposes DIMS, which learns a distribution over diffusion steps by minimizing the expected squared RKHS error between the kernel mean embedding of selected intermediate answers and that of final answers; Prop. 1 decomposes this error into an MMD distribution-mismatch term and a within-trajectory redundancy term, giving a convex QP with a per-step cap, calibrated offline on a small unlabeled set (Ncal=100 prompts, R=20 trajectories by default). At test time one trajectory suffices: M=20 steps are kept (top-M or 75% coverage) and fed to any of eight semantic estimators. On three LLDMs (five variants) and six datasets, DIMS is best or tied-best single-trajectory strategy in 34/40 (estimator, model) blocks, at 66.81 s vs 3.50 s overhead per prompt against 20 i.i.d. generations (Table 1a).
Critical strengths
Clean and genuinely useful formulation: treating the sampling strategy itself as a learnable component of semantic UQ is a novel angle, and the exact decomposition (Prop. 1) plus equivalence to the M-sample target up to a constant (Cor. 1) yield a convex, label-free objective that plugs into any existing semantic estimator unchanged.
Broad and candid evaluation: 5 model variants x 6 datasets x 8 estimators, with ablations on lambda, kernel, encoder, selection rule and calibration budget (stable down to Ncal=10, R=5), transfer experiments, and overhead accounting that includes similarity-model and estimator cost (App. D.4). The paper honestly reports where one trajectory is not enough (EigenVal prefers Full; CoCoA still trails 20 i.i.d. generations by 0.021-0.044 AUROC).
Critical weaknesses
Statistical support for the headline win count is thin: many per-block margins over Full/Random are of the same order as the reported +/-1.96 SE intervals in Table E.1 (+/-0.016-0.020 per cell), and no paired test is provided; averaged over variants, top-M reaches 0.653 vs 0.643 for Full and 0.636 for Random (Table 1a). Together with the large residual gap to the 20-i.i.d. reference for six of eight estimators (0.709 avg vs 0.653/0.655 for DIMS), the "one trajectory is enough" message is really carried by CoCoA and AD.
The headline efficiency claim assumes sequential decoding: App. D.4 charges the reference as 20 sequential decodes and notes batched decoding still costs more than 6x a single trajectory, yet the abstract and intro quote ~66.8 s to 3.5 s without that caveat. The overhead figures are also slightly inconsistent across the paper (intro 66.8/3.5 s; Sec. 5.2 66.7/3.4 s; Table 1a 66.81/3.50 s; Table E.1 66.78 s).
Questions for authors
Could you add paired significance tests (or SEs of the per-block differences) for DIMS vs Full and vs Random? How many of the 34/40 wins survive?
Which overhead figure do you consider deployment-realistic — the sequential ~66.8 s or the batched (>6x single trajectory) number in App. D.4? Would you state this caveat in the abstract?
Why does EigenVal prefer Full (0.621) over top-M (0.616)? Is the redundancy term w^T C w mis-specified for eigenvalue-based scores?
All results use M=20. How sensitive are w* and the selected step sets to M, given the cap set W_M; do weights calibrated at M=20 transfer to, say, M=10?
2 comments
3. Why EigenVal prefers Full (Q3).
We thank the reviewer for highlighting this interesting point that deserves deeper investigation. Our current hypothesis is that the redundancy term is not mis-specified for eigenvalue-based scores, and the cause of EigenVal preferring Full is its sensitivity to the late, converged steps.
First, the redundancy term is not specific to any estimator. The estimator only influences which is selected. EigenVal's , and the resulting selected steps, are typical:
| EigenVal | Eccentricity | DegMat | KLE-Heat | KLE-Matérn | |
|---|---|---|---|---|---|
| median | 2 | 0.85 | 1.5 | 2 | 1.5 |
| mean position of the selected steps, | 0.42 | 0.50 | 0.43 | 0.40 | 0.42 |
and a quick ablation on shows that the redundancy term helps EigenVal:
| EigenVal with DiMS top-M | AUROC | vs Full (0.621) |
|---|---|---|
| =0 (no redundancy term) | 0.590 | -0.031 |
| (label-free on validation split, like in the paper) | 0.616 | -0.005 |
| Oracle: best on test split | 0.629 | +0.008 |
Second, EigenVal’s behavior is not an eigenvalue problem, but rather the consequence of its sensitivity to late steps. Eccentricity is another method that uses the eigenvectors of the same graph Laplacian and does gain from DiMS. Moreover, we notice that the Pearson correlation between AUROC differences Last - Full and DiMS(top-M) - Full is 0.87 over the 8 estimators. EigenVal has the worst Last - Full gap among the 8 uncertainty estimators (-0.058). Our current hypothesis to explain this is that EigenVal acts as a soft count of semantic clusters, but ignores how many steps fall into each cluster, thus suffering from selecting late, converged steps. We acknowledge that we only exhibit a correlation, which may not fully explain EigenVal’s behavior.
Finally, we would like to mention that the statistical support of the claim “EigenVal prefers Full” is also thin. Over the 30 EigenVal runs (5 models, 6 datasets), we have the following table using the same procedure as in our answer to Q1:
| Metric | AUROC | AUROC | PRR | PRR |
|---|---|---|---|---|
| DiMS rule | Top-M vs Full | Cov75 vs Full | Top-M vs Full | Cov75 vs Full |
| Blocks won by DiMS (CI higher than 0) | 2 | 3 | 1 | 1 |
| Tie (and AUROC/PRR > 0) | 10 | 14 | 12 | 17 |
| Tie (and AUROC/PRR < 0) | 14 | 12 | 17 | 11 |
| Blocks lost by DiMS (CI lower than 0) | 4 | 1 | 0 | 1 |
Conclusion. The redundancy term is not mis-specified for eigenvalue-based scores. It is computed without reference to the estimator; it improves EigenVal by +0.026, and Eccentricity, which also uses the Laplacian's spectrum, gains from DiMS. EigenVal's slight preference for Full reflects its strong reliance on early-trajectory diversity (Last − Full = −0.058, the largest drop of all estimators). We will add this analysis to the appendix by the end of the week.
4. Sensitivity to M (Q4).
We agree that a sensitivity analysis on would be valuable to complete our work. We thank the reviewer for pointing this out and giving us the opportunity to disentangle the two roles of the parameter . On the one hand, at calibration time, it defines a cap for the simplex in the quadratic program we solve. On the other hand, at test time, through the top- rule, it defines how many steps we extract from the optimal vector . We therefore study the sensitivity to at these two levels, as well as the transferability across values of used at calibration and test. We report the average AUROC obtained in the following table. Since we evaluate top- with up to 40, we exclude the TriviaQA dataset from the average, as it has only 32 steps, while the other datasets have 128.
| at calibration (rows ) — at test (columns ) | 5 | 10 | 20 | 40 |
|---|---|---|---|---|
| 5 | 0.620 | 0.628 | 0.635 | 0.634 |
| 10 | 0.620 | 0.628 | 0.634 | 0.633 |
| 20 | 0.618 | 0.629 | 0.636 | 0.634 |
| 40 | 0.620 | 0.630 | 0.634 | 0.636 |
It appears that Top- decides the results; the cap during calibration has very little effect on the downstream AUROC. The weights then transfer well to any other at test time. We will add this sensitivity and transfer analysis to the revised version of the paper by the end of the week.
Thank you again. We believe these additions make the statistical claims more rigorous and the efficiency claims more transparent, and we would be glad to continue the discussion.
Dear Reviewer,
First of all, we truly thank you for the thorough and precise review. We thank you for your advice. Your questions pointed to concrete improvements, and we have addressed each one.
1. Statistical support for the win count (Q1).
We agree that the statistical support was thin. In the paper, we only gave a separate confidence interval for each method (Tables E.1 and E.2). This does not show directly whether DiMS beats a baseline on the same prompts, so we ran paired tests as you suggested.
What we did. We used a paired bootstrap over prompts (2,000 resamples). In each resample, we draw a set of test prompts and score all methods on it. We then compute the average difference in AUROC (ΔAUROC) between DiMS and each baseline, with its 95% confidence interval (CI). A block counts as a significant win if the CI is entirely above 0.
We consider 4 cases:
We also computed DeLong's paired significance tests for each block, with small modifications since each block’s AUROC is an average of 6 datasets and 4 LLM judges (for robustness). DeLong’s test confirms the wins obtained with paired prompt bootstrap (29 significant wins against Random, and 22 against Full). We will include the paired significance tests and the CIs of the differences in the revised version of the paper by the end of the week.
2. Deployment-realistic overhead (Q2).
We thank the reviewer for raising this question that is critical to analyse the accuracy-speed trade-off that DiMS offers, as well as for spotting the slight inconsistencies in our figures that were the result of different reruns of the same time measurement experiment. We’ll take care of harmonizing them in a corrected version and send it by the end of the week.
In our opinion, in deployment-realistic scenarios, we may need to assess the uncertainty of a model on several different prompts. Batching these different prompts would be a natural way of optimizing inference and uncertainty computation, especially since our step selection is prompt-independent, so it requires memory writing at a simultaneous time for all prompts in the batch. For i.i.d. sampling, if the M=20 samples for a given prompt are already batched, further batching across different prompts may be limited by memory depending on the GPU.
For this reason, it seems more relevant to us to compute reference independent sampling with sequential decoding rather than batched, as it is more revealing of the deployment setup. But we will reformulate the abstract to highlight that point.