acceptodds
Under review as a conference paper at ICLR 2027

Heterogeneous Uncertainty Along Denoising Traces Signals Hallucinations in Diffusion Language Models

Abstract

Given the rich token-confidence information they contain across positions and time steps, the denoising traces of diffusion language models offer a natural foundation for uncertainty quantification and hallucination detection. Yet, existing detection methods either derive from autoregressive models—failing to capture the full extent of this information—or process it using heavyweight neural architectures. In this paper, we show that hallucinations are better predicted by the heterogeneity of uncertainty along this trace than by its average level. In particular, large variations in token distribution entropy over denoising steps provide a strong discriminative signal. A Markovian model of token-wise convergence links this pattern to slower and more heterogeneous convergence in hallucinated answers. Building on these findings, we introduce Drifty: a lightweight detector combining 37 closed-form trace features with regularized logistic regression. Across three diffusion language models and three closed-book question-answering benchmarks, Drifty outperforms all evaluated baselines in every model–dataset pair, improving over Semantic Entropy by up to 14.5 ROC-AUC points and 16 PR-AUC points. Moreover, Drifty uses only the trace of a single deterministic generation, with feature extraction and classification requiring roughly half a second of CPU time per 100 samples.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Uncertainty Quantification, Hallucinations, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VUivOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRIFTY, a lightweight hallucination detector for diffusion language models that extracts 37 statistical features from denoising traces and uses regularized logistic regression for classification. The authors argue that cross-token uncertainty heterogeneity is more informative than average uncertainty and provide a Markovian convergence analysis to explain this phenomenon. Experiments across three diffusion models and three QA benchmarks show competitive detection performance at low post-processing cost.

My main concern is that the evidence does not fully establish the proposed explanation. The experiments do not directly isolate the contribution of uncertainty heterogeneity from mean-level features, while the theoretical analysis makes stronger claims about token convergence and temporal persistence than the reported results support.

Critical strengths
  1. Strong detection performance with low post-processing cost.
  2. Reasonably comprehensive experimental evaluation.
Critical weaknesses
  1. Insufficient evidence for the central claim that uncertainty heterogeneity is more informative than average uncertainty. The paper mainly supports this claim through feature–label correlations and detection results. However, Figure 6 only reports individual feature correlations on Dream, while the ablation baseline in Appendix D already combines level, dispersion, and shape features. Without a controlled comparison of level-only and heterogeneity-only features, the claimed superiority of uncertainty heterogeneity is not directly established.
  2. The proposed convergence mechanism is not sufficiently validated. The authors attribute greater uncertainty heterogeneity in hallucinated responses to more dispersed token convergence rates. However, the proposed model also contains token-specific boundary constants that can independently affect entropy variation. Moreover, Figure 4 only compares class-averaged entropy decay, rather than the distributions of per-token convergence rates. The experiments therefore do not establish that heterogeneous convergence rates are responsible for the observed detection signal.
Questions for authors

Can the authors provide a controlled comparison of level-only, heterogeneity-only, and combined features using the same classifier to directly verify the central claim?

Can the authors compare the distributions of estimated per-token convergence rates between correct and hallucinated responses, while accounting for the token-specific boundary constants in Equation (2)?

Do the proposed uncertainty features remain effective under native sampling strategies, particularly DiffusionGemma's entropy-bounded sampler, rather than the modified deterministic sampler used in the experiments?

2 comments
AuthorsOpenReview ↗

(Part 1/2)

We thank Reviewer VUiv for proposing direct tests of the heterogeneity claim, the convergence mechanism and the native sampler. We ran these analyses. They support the claim for absorbing-state models. For DiffusionGemma they show that the convergence-rate account does not carry over cleanly, and we now say so explicitely in the main text.

Summary of Manuscript Changes

  • Section 3.1: the uncertainty mask mt,dm_{t,d} is defined for both absorbing-state and uniform-state models.
  • Section 3.3, Appendix F (Table 12, Figure 7): per-token τ^d\hat\tau_d distributions (ECDFs, KS tests), per-sequence dispersion with and without the ≥4\geq 4-step restriction, and the τ-only and C-only counterfactuals. DiffusionGemma results are reported in full, including the ≥4\geq 4-step reversal.
  • Sections 3.3 and 6: the convergence-rate account is stated as established for absorbing-state models only.
  • Section 5.2, Tables 13 and 14: controlled feature-family ablations per model, with effect sizes for partial correlations.
  • Appendix F: sign consistency across the three architectures for all nine (32,16)(32,16) configurations, extending Figure 6 (Dream only).

W1 & Q1: Level-Only, Heterogeneity-Only and Combined Features Under One Classifier

Nine-pair and per-model means at 32 tokens / 16 steps (ROC / PR), with the same regularised logistic regression and 80/20 splits:

Feature setDreamLLaDADiffusionGemmaNine pairs
Level-only (8)0.745 / 0.7910.778 / 0.8570.758 / 0.7520.761 / 0.800
V‾(t)\overline{V}(t) alone (1, no fitting)0.780 / 0.8390.798 / 0.8810.753 / 0.7410.777 / 0.820
V(t)V(t)-curve descriptors (8)0.791 / 0.8490.805 / 0.8810.753 / 0.7470.783 / 0.826
Pure dispersion + shape + rates (25)0.820 / 0.8670.822 / 0.8940.762 / 0.7660.801 / 0.842
Non-level (29)0.822 / 0.8670.819 / 0.8930.775 / 0.7780.806 / 0.846
Full DRIFTY (37)0.828 / 0.8710.824 / 0.9000.776 / 0.7790.809 / 0.850
  • On Dream and LLaDA, heterogeneity-only features beat level-only by 4.4 to 7.5 ROC points. On DiffusionGemma the two families are tied (0.762 against 0.758), and level-only is slightly better on NQ (0.741 against 0.737 for Non-level). Section 5.2 now states this.
  • Dispersion features have the same positive sign with hallucination in all three architectures (Appendix F). For V‾(t)\overline{V}(t) the Pearson correlations across datasets are +0.39 to +0.57 (Dream), +0.36 to +0.54 (LLaDA) and +0.32 to +0.46 (DiffusionGemma). Appendix F covers all nine configurations, not just Dream.
AuthorsOpenReview ↗

(Part 2/2)

W2 & Q2: Convergence Rates versus Boundary Constants

Figure 4 shows class-averaged entropy decay; Figure 7 compares per-token distributions. For each non-padding token we fit ln⁡Ht,d=−t/τd+ln⁡Cd\ln H_{t,d}=-t/\tau_d+\ln C_d over the steps where it is still masked. We compare the τ^d\hat\tau_d distributions and build two counterfactual trajectories per sequence: τ-only keeps each token's rate and sets its constant to the sequence mean; C-only does the reverse. These counterfactuals analyse fitted parameters. They are not interventions on the model.

Absorbing-state models (Dream, LLaDA). Hallucinated answers have a heavier upper tail of slow-resolving tokens. On TriviaQA the 75th percentile of τ^d\hat\tau_d rises from 10.2 to 18.2 steps on Dream and from 10.0 to 16.4 on LLaDA. The standard deviation of τ^d\hat\tau_d across tokens within a sequence is 35–65% larger on hallucinated sequences over all non-padding tokens, and 36–69% larger when restricted to tokens with at least four active steps. The counterfactuals agree. C-only falls to chance (0.504–0.531 ROC), while τ-only keeps 0.693–0.767 (empirical 0.743–0.837). On these models the signal comes from rate heterogeneity rather than from boundary constants.

DiffusionGemma. The picture differs, and we report it in full. The per-token τ^d\hat\tau_d distributions are nearly identical (median 1.90 for correct answers against 1.92 for hallucinated ones on TriviaQA). The differences are significant but small (KS D≈0.03D\approx0.03). Over all tokens, dispersion is not larger on hallucinated sequences (p=0.73p=0.73–0.860.86). Restricted to tokens with at least four active steps, it is significantly smaller on hallucinated sequences for TriviaQA (5.78 against 6.24 for correct answers) and for NQ (4.34 against 4.51, p=0.008p=0.008), and not significant for HotpotQA (p=0.45p=0.45). The two counterfactuals are not separable: τ-only reaches 0.697–0.730 ROC and C-only 0.656–0.746, with C-only higher on TriviaQA. Uniform-state canvas refinement therefore does not support the convergence-rate account, and Sections 3.3 and 6 now say so.

Q3: Native Sampler and Stochastic Decoding

With one fixed threshold, ϵ=0.1\epsilon=0.1, on all three DiffusionGemma datasets, ROC-AUC changes by +0.003 to +0.009. That is within the bootstrap standard deviation, so we report the native mask as comparable to the fixed schedule (Table 17b). At T=0.8T=0.8, DRIFTY reaches 0.824 and 0.898 ROC on Dream TriviaQA and Dream HotpotQA. We test the transfer on Dream: a classifier trained on deterministic traces transfers zero-shot to stochastic traces with 0.842 and 0.827 (Table 17a). See also our response to Reviewer RhBg (in their Q3).

We renew our thanks to Reviewer VUiv for their time reviewing Drifty !

Reviewer_RhBgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DRIFTY, a lightweight hallucination detector for diffusion language models. The main observation is that hallucinated answers show greater variation in token uncertainty throughout denoising, rather than simply having higher average uncertainty.

The authors extract 37 statistical features from denoising trajectories, including entropy variance, convergence rates, and temporal patterns, and use regularized logistic regression for classification. Experiments on LLaDA, Dream, and DiffusionGemma across three QA benchmarks show competitive detection performance with low post-processing cost.

Critical strengths

Lightweight and practical detection.

Using simple statistical features with logistic regression avoids the complexity of neural trajectory detectors. The reported detection performance and low post-processing cost are promising.

Broad experimental evaluation.

The paper evaluates three diffusion language models, multiple QA datasets, different generation lengths, and cross-dataset transfer. The human validation of factuality labels also strengthens the experimental setup.

Critical weaknesses
  1. Limited novelty and missing closely related work.

The central idea of using uncertainty dynamics during diffusion decoding for hallucination detection has already been explored in several studies.

OSCAR (Shah et al., 2026) investigates token commitment uncertainty and hallucination localization. TRE (Weng et al., 2026) already uses spatial and temporal entropy from a single generation for lightweight hallucination detection. DeMTS (Zhang et al., 2026) studies token-time uncertainty patterns and inconsistent convergence.

Even though DRIFTY uses a different feature-based formulation, its broader motivation substantially overlaps with these methods. The absence of these papers, particularly TRE and DeMTS, and the lack of direct comparisons make it difficult to establish the significance of the paper.

  1. Insufficient validation of uncertainty heterogeneity.

The paper argues that cross-token entropy variance reflects different convergence speeds between factual and hallucinated responses. This variance may also be influenced by the remasking schedule, response length, and the number of remaining masked tokens.

The current experiments do not fully isolate these effects or establish how much improvement comes specifically from the proposed variance features. Furthermore, the evaluation focuses on deterministic decoding, leaving the robustness of the approach under other decoding strategies unclear.

Questions for authors

Novelty: How does DRIFTY differ from OSCAR, TRE, and DeMTS? Can the authors provide direct comparisons of detection performance and computational cost?

Feature contribution: How much improvement comes specifically from cross-token entropy variance? Can authors provide separate ablations controlling for average entropy, masking behavior, and response length?

Generalization: Does DRIFTY remain effective under stochastic decoding and native DiffusionGemma sampling? Can the authors also report statistical variability across multiple runs?

2 comments
AuthorsOpenReview ↗

(Part 1/2)

We would like to thank Reviewer RhBg for their time reviewing our paper and for pointing us to OSCAR, TRE and DeMTS, and for asking for controlled ablations, decoding robustness and variability estimates. We have integrated all three works into Section 2.3 and ran the requested experiments.

Summary of Manuscript Changes

  • Section 2.3: Added discussions of OSCAR (position-level cross-chain Shannon entropy over N=8N=8 randomized reveal orders), TRE (step-weighted commitment-step entropy) and DeMTS (multivariate time-series network with step-wise variance normalization).
  • Section 5.1, Tables 3 and 10: TRE is implemented from its closed-form definition at both schedules. Semantic Entropy is presented as a multi-sample output-level baseline (N=5N=5 stochastic generations clustered by NLI).
  • Section 5.2, Tables 4, 13 and 14: controlled feature-family ablations, confounder controls and effect sizes.
  • Section 5.2, Table 17b: native-mask results reported as comparable to the fixed schedule.
  • Appendix G, Table 16: variability estimates.
  • Section 6: protocol differences with DeMTS and OSCAR, and the cost comparison.

W1 & Q1: Comparison with OSCAR, TRE and DeMTS

TRE (implemented following the preprint to the best of our abilities). TRE (Weng et al., 2026) is training-free and needs a single generation. We implemented its closed-form definition, TRE=∑t=1TtT∑d∈TtgHt,d\mathrm{TRE}=\sum_{t=1}^{T}\frac{t}{T}\sum_{d\in\mathcal{T}^g_t}H_{t,d}, where Ttg={d:mt,d=1∧mt+1,d=0}\mathcal{T}^g_t=\{d: m_{t,d}=1 \land m_{t+1,d}=0\} is the set of positions unmasked (or first accepted, for DiffusionGemma) at step tt, and evaluated it on all nine pairs at both schedules. DRIFTY is above TRE in every cell at both schedules (18 of 18 each). Nine-pair means are 0.732 / 0.769 for TRE and 0.809 / 0.850 for DRIFTY at 32 tokens / 16 steps, and 0.649 / 0.692 and 0.797 / 0.838 at 256 / 128. The single feature V‾(t)\overline{V}(t) (0.777 / 0.820 at 32 / 16) already beats TRE. TRE scores each position only at its unmasking step, so it cannot measure dispersion among positions that are still masked. TRE and V‾(t)\overline{V}(t) take under 0.01 s per 100 samples on CPU; DRIFTY takes 0.43–0.49 s.

DeMTS (reported results only). DeMTS (Zhang et al., 2026) divides out the cross-token standard deviation σt\sigma_t as a schedule effect (H^i,t=(Hi,t−μt)/(σt+ϵ)\widehat{H}_{i,t}=(H_{i,t}-\mu_t)/(\sigma_t+\epsilon)) and models trajectories with a multi-module network over intermediate hidden-state embeddings ei,t∈Rdmodel\mathbf{e}_{i,t}\in\mathbb{R}^{d_{\text{model}}}. Our results point the other way: σt2=V(t)\sigma_t^2=V(t) over masked positions is itself the primary signal, with single-feature AUROC between 0.723 and 0.837 across the nine (32, 16) pairs. DeMTS has no public code and requires full intermediate hidden states across all steps, so we could not run it directly. Its cross-task table (trained on TriviaQA, tested on HotpotQA: 82.2 AUROC; reverse direction: 74.1) uses Qwen3-8B labels and generation lengths of 64 and 128 tokens, and does not state its backbone. For reference only, DRIFTY trained on TriviaQA gives 0.793 on HotpotQA with LLaDA (0.799 with Dream), and 0.838 in the reverse direction (0.835 with Dream). The protocols differ, so these numbers are not directly comparable.

OSCAR (not reproduced). OSCAR (Shah et al., 2026) flags uncertain positions with cross-chain Shannon entropy over N=8N=8 parallel denoising chains with randomized reveal orders, and then remasks and re-denoises flagged spans (optionally with RAG). The chains are batched into one forward pass per step, and OSCAR reports about 1.2× wall-clock overhead for this detection stage (1.3× end-to-end). OSCAR has no public code, and differs from our multi-sample Semantic Entropy baseline (five sampled answers at T=0.5T=0.5, clustered by bidirectional DeBERTa NLI). Its detection stage requires eight times the forward-pass compute of a single generation, whereas DRIFTY needs one generation plus 0.43–0.49 s of CPU time per 100 samples.

Semantic Entropy (Table 3). Single-pass DRIFTY beats multi-sample Semantic Entropy (N=5N=5) in all 18 cells. Nine-pair means are 0.745 / 0.749 for Semantic Entropy and 0.809 / 0.850 for DRIFTY, a gain of +6.4 ROC and +10.1 PR points. The largest gains are on Dream NQ (+14.5 ROC, +16.1 PR). The 188–459 s per 100 prompts reported for Semantic Entropy in Table 3 measures the total wall-clock time on 2 GPUs required beyond the primary generation—namely, generating the N=5N=5 stochastic answers at T=0.5T=0.5 and running pairwise DeBERTa-v2-xlarge-MNLI entailment clustering—whereas single-trace methods require no additional generations.

AuthorsOpenReview ↗

(Part 2/2)

W2 & Q2: Heterogeneity versus Average Entropy, Masking and Length

Under the same regularized logistic regression and splits (Tables 4 and 13), the eight V(t)V(t)-curve descriptors beat the eight level features by 2.2 ROC and 2.6 PR points. Removing the eight mean-level features from the full model costs 0.004 ROC (with p=0.21p=0.21). Confounder controls are in Table 14 and in our response to Reviewer moRg (in W1). In brief:

  • Average entropy. After partialling out the mean masked entropy, the residual association stays significant in all nine configurations. For DiffusionGemma, it is weak (partial correlations 0.16, 0.05 and 0.06 on TriviaQA, NQ and HotpotQA respectively).
  • Masking. On our primary schedule, the number of masked positions at step tt is fixed by DD and TT, so it cannot vary between samples at a given step. The native non-monotonic DiffusionGemma mask is evaluated separately (see Q3).
  • Response length. Effective length alone is near chance (with AUROC 0.500–0.602). Padding-excluded V‾(t)\overline{V}(t) reaches 0.723–0.821.
  • DiffusionGemma. Level and dispersion are tied there. The pure dispersion + shape + rates set reaches 0.762 ROC against 0.758 for level-only, and mean entropy absorbs 60–77% of the raw V‾(t)\overline{V}(t) correlation. The claim that heterogeneity carries the signal therefore holds for absorbing-state models (Dream, LLaDA) and is weaker for DiffusionGemma. Sections 5.2 and 6 now state this.

Q3: Stochastic Decoding, Native Sampler and Variability

  • Stochastic decoding (T=0.8T=0.8, Table 17a). DRIFTY reaches 0.824 and 0.898 ROC on Dream TriviaQA and Dream HotpotQA, against 0.759 and 0.742 for Level-only. We test the transfer on Dream: a classifier trained on deterministic traces transfers zero-shot to stochastic traces with 0.842 and 0.827 ROC.
  • Native DiffusionGemma sampler (Table 17b). With one fixed threshold, ϵ=0.1\epsilon=0.1, on all three datasets, ROC-AUC changes by +0.009, +0.005 and +0.003 (TriviaQA, NQ, HotpotQA), and PR-AUC by +0.008, +0.003 and −0.004. These changes are within the bootstrap standard deviation (about 0.02). We therefore report the native rule as comparable to the fixed schedule, not better. At ϵ=0.5\epsilon=0.5, NQ ROC is 0.735 against 0.737.
  • Variability (Table 16). Generation at T=0T=0 is deterministic, so the primary setting has no repeated generations. We report two other sources of variability. Test-split bootstrap standard deviations are 0.016–0.023 ROC, with 95% intervals such as [0.819, 0.883] for DRIFTY on Dream TriviaQA. Ten repeated stratified 80/20 splits give ROC standard deviations of 0.006–0.029 (PR up to 0.042). The T=0.8T=0.8 results are single runs over 1,260 stochastic traces.

Finally, we would like to thank you again for your review and we hope we have addressed most of your concerns.

Reviewer_moRgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination detection in diffusion language models (d-LLMs). The central claim is that the heterogeneity of uncertainty across tokens during denoising—not its average level—best distinguishes hallucinated from correct answers. The authors propose DRIFTY, a lightweight detector that distills the denoising trace into 37 closed-form statistics (dispersion, shape, and convergence rates) fed to a regularized logistic regression, avoiding neural architectures entirely. The method is evaluated on three d-LLMs and three QA benchmarks against both transposed autoregressive baselines and the neural TraceDet detector.

Critical strengths

S1: Identifies a real gap: AR detectors discard d-LLM trace information; neural trace methods are too heavy.

S2: Cross-token dispersion (V(t)) separates hallucinations from correct answers more effectively than mean-level statistics, with a plausible Markovian convergence model explaining the mechanism.

S3: Practical efficiency. Single deterministic generation, CPU-only, sub-second scoring—deployable where neural detectors are not.

Critical weaknesses

W1: The core claim—that trace heterogeneity signals hallucinations—rests on observed correlations. No interventional experiments (e.g., noise injection, forced early commitment) establish causality. Confounders like question difficulty or answer length could drive both V(t) elevation and hallucination propensity; the paper offers no controls to rule these out.

W2: TraceDet is evaluated with untuned default hyperparameters, while DRIFTY undergoes full 5-fold CV. This asymmetric protocol likely underestimates TraceDet.

W3: The 37-feature set lacks systematic ablation. Top features are highly correlated descriptors of the same V(t) phenomenon; many tail features contribute marginally. The paper never shows that a reduced subset performs significantly worse, leaving open whether this elaborate feature engineering is necessary or optimal.

W4: Evaluation is restricted to short-form closed-book QA under deterministic decoding. Applicability to long-form generation, stochastic sampling, or non-QA tasks is asserted but not demonstrated.

Questions for authors

Q1: The appendix ablation (Baseline vs. Markov vs. DRIFTY) shows Markov features add incremental value, but does not address redundancy within the 37-feature set. What is the performance of a minimal subset? Is the full feature set necessary, or does a smaller subset suffice?

Q2: How does DRIFTY perform on longer, more complex generations (e.g., multi-hop reasoning, summarization, code generation) where hallucinations may be localized to specific spans? Does the V(t) signal remain discriminative, or does aggregating token-level uncertainties dilute the hallucination signature?

2 comments
AuthorsOpenReview ↗

(Part 1/2)

We deeply thank Reviewer moRg for their time and careful review and for recognising the Markovian mechanism and the CPU-only cost. The main concern, that our evidence is observational, is correct. We now state this explicitly and have removed causal wording that the evidence does not support.

Summary of Manuscript Changes

  • Sections 1, 3.2–3.4 and 7: causal and overstated wording removed. The Markovian model is presented as a mechanism consistent with the data, for absorbing-state models.
  • Section 5.2, Table 14 in appendix: within-question paired tests, leave-one-model-out difficulty strata and partial correlations, reported with effect sizes.
  • Section 5.1: scope of TraceDet tuning, and the 256-token comparison with TraceDet.
  • Section 5.2, Tables 4 and 13: composition of the Top-5 row, and the source of the feature ranking.
  • Section 6: observational scope, weaker residual coupling on DiffusionGemma, the 256-token prompt protocol, and non-QA scope.

W1: Observational Evidence and Confounders

Our evidence is indeed solely observational. We analyse recorded traces and do not intervene on decoding. We have not run interventions such as noise injection or forced early commitment, and Section 6 lists this as a limitation. The mechanism claims are restricted to absorbing-state models (see also our response to Reviewer VUiv).

For the three confounders the reviewer names:

  • Question difficulty. On the 2,735 questions where models disagree (at least one answers correctly and at least one hallucinates), model-standardised V‾(t)\overline{V}(t) is higher on the hallucinated answers: TriviaQA Δ=+0.441\Delta=+0.441, Natural Questions Δ=+0.231\Delta=+0.231, HotpotQA Δ=+0.225\Delta=+0.225. Within leave-one-model-out difficulty strata, V‾(t)\overline{V}(t) remains discriminative (Dream on TriviaQA: AUROC 0.783, 0.731 and 0.734 for easy, medium and hard).
  • Answer length. Effective non-padding length alone is near chance (AUROC 0.500–0.602). Padding-excluded V‾(t)\overline{V}(t) reaches 0.723–0.821.
  • Mean masked entropy. Partialling out length, mean masked entropy and difficulty jointly keeps V‾(t)\overline{V}(t) significantly associated with hallucination in all nine configurations (with at least p≤0.038p\le0.038). The effect size depends on the model. On Dream and LLaDA the partial correlation is 0.13–0.33. On DiffusionGemma it is 0.16, 0.05 and 0.06 (for TriviaQA, NQ, HotpotQA respectively), and partialling out mean entropy removes 60–77% of the raw correlation (14–35% on Dream and LLaDA). The residual DiffusionGemma signal is weak, and Sections 5.2 and 6 now say so explicitely.

W2: TraceDet Hyperparameters

We ran a 12-configuration validation grid for TraceDet on the same splits: learning rate ∈{10−4,2×10−4,5×10−4}\in\{10^{-4},2\times10^{-4},5\times10^{-4}\}, depth L∈{2,3}L\in\{2,3\} and information-bottleneck weight β∈{0.05,0.10}\beta\in\{0.05,0.10\}. Each configuration is trained with early stopping on a stratified 10% validation split of the training data (roughly 170–200 examples per pair). The configuration with the best validation AUROC is then evaluated on the test split (Table 15).

Tuning does not improve TraceDet. Its nine-pair mean is 0.777 / 0.813 (ROC / PR) tuned, against 0.779 / 0.816 with defaults, and the tuned model has lower ROC on 7 of 9 pairs. This is consistent with noisy selection on a small validation split. DRIFTY reaches 0.809 / 0.850. Against tuned TraceDet, DRIFTY matches or exceeds it in 16 of 18 cells (15 of 18 against the default).

The scope is limited. The grid varies three hyperparameters. Dropout, weight decay, feed-forward width, positional-encoding dimension and batch size stay at the "official" defaults. TraceDet was tuned only at the 32 tokens configuration.

AuthorsOpenReview ↗

(Part 2/2)

W3 & Q1: Redundancy and Minimal Subsets

Redundancy is hard to avoid with our complete set of features. The strongest univariate features are variants of V‾(t)\overline{V}(t). The variance-curve AUC, for example, has r=0.998r=0.998 with V‾(t)\overline{V}(t). Regularisation handles this, but a smaller set could still suffice. It does not, and the advantage does not come from the level features. Nine-pair means at 32 tokens / 16 steps, all with the same logistic regression:

Feature setROC / PR
Level-only (8)0.761 / 0.800
Top-1: V‾(t)\overline{V}(t) alone (no fitting)0.777 / 0.820
V(t)V(t)-curve descriptors (8)0.783 / 0.826
Pure dispersion + shape + rates (25)0.801 / 0.842
Non-level (29)0.806 / 0.846
Full DRIFTY (37)0.809 / 0.850
  • Full DRIFTY exceeds Top-1 by 3.2 ROC points (paired tt-test p=0.001p=0.001), Top-3 by 3.1 (p=0.001p=0.001), Top-5 by 3.0 (p=0.003p=0.003) and Level-only by 4.9 (p≈0.004p\approx0.004). Paired tests use the nine pairs.
  • Dimension-matched: the eight V(t)V(t)-curve descriptors beat eight level features by 2.2 ROC and 2.6 PR points.
  • Removing the eight mean-level features from the full model costs 0.004 ROC (p=0.21p=0.21).
  • Stability selection (from Meinshausen and Bühlmann, 2010; 200 bootstrap subsamples on the training split, πthr=0.6\pi_{\text{thr}}=0.6, after ∣r∣>0.95|r|>0.95 correlation prefiltering) keeps 7.7 features per configuration on average, with 0.801 ROC / 0.868 PR. All features give 0.814 / 0.876 over the 24 configurations in Appendix G (Dream and LLaDA ×\times 3 datasets ×\times 4 schedules: (32,16)(32,16), (32,32)(32,32), (64,32)(64,32) and (64,64)(64,64)).

Ranking source. The Top-k rows rank features by univariate ∣r∣|r| computed on the full Dream dataset, including its test folds. Using test labels in selection can only make these rows look better, so the gap to Full DRIFTY is, if anything, underestimated. V‾(t)\overline{V}(t) is the statistic of Eq. (1), not a choice made from the ranking. The Top-5 row is {VarMaskedEntropyAcrossTokens, AUCVarMaskedEntropyCurve, VarLogProbMaskedAcrossTokens, ArgmaxVarMaskedEntropyCurve, AlphaEntropyAvg}. It is not the strict top-5 by ∣r∣|r|, because MaxVarMaskedEntropyCurve ranks fourth. Tables 4 and 13 now state this composition.

W4 & Q2: Longer Generations, Stochastic Decoding and Non-QA Tasks

We answer this in part. Stochastic decoding and 256-token generation are covered below. We did not evaluate summarisation or code generation, and Section 6 says so.

  • Stochastic decoding (T=0.8T=0.8, Table 17a). Each 2,100-prompt mixed-temperature run has 840 deterministic (T=0.0T=0.0) and 1,260 stochastic (T=0.8T=0.8) traces. After removing unclear verdicts, 838 and 1,256 remain on TriviaQA, and 837 and 1,246 on HotpotQA. In-domain evaluation uses an 80/20 split of the stochastic pool (1,004 train, 252 test on TriviaQA). Transfer is scored on all stochastic traces, so the two protocols are not directly comparable. Output-level perplexity and LN-entropy stay near chance (0.54–0.59 ROC). DRIFTY reaches 0.824 and 0.898 ROC on Dream TriviaQA and Dream HotpotQA. We test the transfer on Dream: a classifier trained only on the deterministic traces transfers zero-shot with 0.842 and 0.827 ROC. Non-level features beat Level-only by 5.9 and 15.3 ROC points.
  • Longer generations (256 tokens, Tables 10 and 13). Nine-pair means are 0.704 ROC for Level-only, 0.796 for Non-level and 0.797 for Full DRIFTY. On DiffusionGemma, global averaging dilutes single-feature V‾(t)\overline{V}(t) (0.700, 0.540 and 0.587 ROC on TriviaQA, NQ and HotpotQA). Variance over 32-token windows recovers 0.769, 0.618 and 0.642 (Table 17c). This windowed statistic is a diagnostic, not one of the 37 features. On Dream and LLaDA, restricting variance to still-masked positions adds 6.9–22.4 ROC points, because padding dominates the unmasked average.
  • Where DRIFTY is weaker. At 256 tokens, untuned TraceDet is competitive. DRIFTY matches or exceeds it in 12 of 18 cells, and TraceDet is ahead on Dream HotpotQA, LLaDA TriviaQA and LLaDA NQ (Table 10). Section 5.1 now says so. DiffusionGemma's 256-token answers fill the canvas (Leff=256L_{\text{eff}}=256): our concise-answer instruction (" please answer briefly") is applied only at D∈{32,64}D\in\{32,64\} and omitted at D=256D=256 to allow unconstrained long-form generation, so verbosity partly reflects prompt format. Section 6 now notes this.

For very long generations (like reasonning traces over multiple canvas), we do expect that some tokens lose their uncertainty markers because the model has decided its answer at earlier stages of the generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.