- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DRIFTY, a lightweight hallucination detector for diffusion language models that extracts 37 statistical features from denoising traces and uses regularized logistic regression for classification. The authors argue that cross-token uncertainty heterogeneity is more informative than average uncertainty and provide a Markovian convergence analysis to explain this phenomenon. Experiments across three diffusion models and three QA benchmarks show competitive detection performance at low post-processing cost.
My main concern is that the evidence does not fully establish the proposed explanation. The experiments do not directly isolate the contribution of uncertainty heterogeneity from mean-level features, while the theoretical analysis makes stronger claims about token convergence and temporal persistence than the reported results support.
Critical strengths
- Strong detection performance with low post-processing cost.
- Reasonably comprehensive experimental evaluation.
Critical weaknesses
- Insufficient evidence for the central claim that uncertainty heterogeneity is more informative than average uncertainty. The paper mainly supports this claim through feature–label correlations and detection results. However, Figure 6 only reports individual feature correlations on Dream, while the ablation baseline in Appendix D already combines level, dispersion, and shape features. Without a controlled comparison of level-only and heterogeneity-only features, the claimed superiority of uncertainty heterogeneity is not directly established.
- The proposed convergence mechanism is not sufficiently validated. The authors attribute greater uncertainty heterogeneity in hallucinated responses to more dispersed token convergence rates. However, the proposed model also contains token-specific boundary constants that can independently affect entropy variation. Moreover, Figure 4 only compares class-averaged entropy decay, rather than the distributions of per-token convergence rates. The experiments therefore do not establish that heterogeneous convergence rates are responsible for the observed detection signal.
Questions for authors
Can the authors provide a controlled comparison of level-only, heterogeneity-only, and combined features using the same classifier to directly verify the central claim?
Can the authors compare the distributions of estimated per-token convergence rates between correct and hallucinated responses, while accounting for the token-specific boundary constants in Equation (2)?
Do the proposed uncertainty features remain effective under native sampling strategies, particularly DiffusionGemma's entropy-bounded sampler, rather than the modified deterministic sampler used in the experiments?
2 comments
(Part 2/2)
W2 & Q2: Convergence Rates versus Boundary Constants
Figure 4 shows class-averaged entropy decay; Figure 7 compares per-token distributions. For each non-padding token we fit over the steps where it is still masked. We compare the distributions and build two counterfactual trajectories per sequence: τ-only keeps each token's rate and sets its constant to the sequence mean; C-only does the reverse. These counterfactuals analyse fitted parameters. They are not interventions on the model.
Absorbing-state models (Dream, LLaDA). Hallucinated answers have a heavier upper tail of slow-resolving tokens. On TriviaQA the 75th percentile of rises from 10.2 to 18.2 steps on Dream and from 10.0 to 16.4 on LLaDA. The standard deviation of across tokens within a sequence is 35–65% larger on hallucinated sequences over all non-padding tokens, and 36–69% larger when restricted to tokens with at least four active steps. The counterfactuals agree. C-only falls to chance (0.504–0.531 ROC), while τ-only keeps 0.693–0.767 (empirical 0.743–0.837). On these models the signal comes from rate heterogeneity rather than from boundary constants.
DiffusionGemma. The picture differs, and we report it in full. The per-token distributions are nearly identical (median 1.90 for correct answers against 1.92 for hallucinated ones on TriviaQA). The differences are significant but small (KS ). Over all tokens, dispersion is not larger on hallucinated sequences (–). Restricted to tokens with at least four active steps, it is significantly smaller on hallucinated sequences for TriviaQA (5.78 against 6.24 for correct answers) and for NQ (4.34 against 4.51, ), and not significant for HotpotQA (). The two counterfactuals are not separable: τ-only reaches 0.697–0.730 ROC and C-only 0.656–0.746, with C-only higher on TriviaQA. Uniform-state canvas refinement therefore does not support the convergence-rate account, and Sections 3.3 and 6 now say so.
Q3: Native Sampler and Stochastic Decoding
With one fixed threshold, , on all three DiffusionGemma datasets, ROC-AUC changes by +0.003 to +0.009. That is within the bootstrap standard deviation, so we report the native mask as comparable to the fixed schedule (Table 17b). At , DRIFTY reaches 0.824 and 0.898 ROC on Dream TriviaQA and Dream HotpotQA. We test the transfer on Dream: a classifier trained on deterministic traces transfers zero-shot to stochastic traces with 0.842 and 0.827 (Table 17a). See also our response to Reviewer RhBg (in their Q3).
We renew our thanks to Reviewer VUiv for their time reviewing Drifty !
(Part 1/2)
We thank Reviewer VUiv for proposing direct tests of the heterogeneity claim, the convergence mechanism and the native sampler. We ran these analyses. They support the claim for absorbing-state models. For DiffusionGemma they show that the convergence-rate account does not carry over cleanly, and we now say so explicitely in the main text.
Summary of Manuscript Changes
W1 & Q1: Level-Only, Heterogeneity-Only and Combined Features Under One Classifier
Nine-pair and per-model means at 32 tokens / 16 steps (ROC / PR), with the same regularised logistic regression and 80/20 splits: