acceptodds
Under review as a conference paper at ICLR 2027

Hallucination Detection in Diffusion Language Models via Sparse Token Informativeness

Abstract

Diffusion large language models (DLMs) have recently emerged as a promising alternative to autoregressive LLMs, with recent systems scaled to as many as 100B-parameter. Yet hallucination in DLMs remains underexplored: we still lack an interpretable understanding of where factual errors emerge during remasking, undermining their reliability in real-world applications. Existing DLM-specific detectors recover richer denoising evidence by training supervised models over selected sub-traces or structured token-interaction graphs. We identify an Early Factual Commitment phenomenon in DLMs: factual responses tend to become distinguishable before generation is complete, as decisive token commitments reduce uncertainty early in the denoising trajectory. Motivated by this observation, we separate each intermediate state into two views: the already committed stable context and the remaining masked positions that encode predictive uncertainty. For each token commitment, we measure its predictive information gain by tracking how it changes uncertainty over these two views. Extensive experiments show that sparse commit-level uncertainty signals provide effective yet interpretable hallucination detection while requiring no auxiliary verifier, repeated sampling, dense trajectory modeling, or learned detector architecture.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination Detection, Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x4XzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies hallucination detection in diffusion language models by considering token commitment and changes in uncertainty. The paper observes Early Factual Commitment and proposes two simple detection signals named Late Mask Entropy (LME) and Token Informativeness (TI). Experiments are conducted on LLaDA, Dream, and DiffusionGemma to evaluate the performance of the proposed method. The results show good AUROC performance without training an additional neural detector.

Critical strengths
  • Late Mask Entropy is simple and shows good performance on LLaDA and Dream. It achieves an AUROC of 88.26 on HotpotQA.

  • The experiments include different diffusion models and remasking strategies. The results show how uncertainty changes differently across decoders.

Critical weaknesses
  • The paper claims that Informativeness measures information gain. However, the proposed definition does not measure the information gain defined in the paper. Equation 2 uses the same remaining masked positions, while Equation 4 subtracts mean entropies over different masked sets. Removing committed tokens from the average adds an extra term to the score. Therefore, a commitment can have a high Informativeness score even if the predictions of all remaining tokens stay the same.

  • TRE already uses token-level entropy and temporal weighting from a single diffusion decoding trajectory to detect hallucinations without training. The proposed LME and TI use different uncertainty signals, but the paper does not compare with TRE to show the performance gains. Table 2 shows that the simpler LME detector outperforms TI in four of six settings on LLaDA and four of six settings on Dream. TI is more robust under random remasking. However, the paper does not compare its Top-5% selection with random or entropy-matched step selection while keeping the final-entropy term the same.

  • The proposed method has a different meaning when applied to DiffusionGemma. This model does not use the same masked-token commitment process. Therefore, the paper uses entropy at fixed answer positions in place of masked-region entropy. However, this does not follow the commitment-level definition used to motivate TI. The performance on DiffusionGemma may come from general changes in entropy, and does not clearly show the effect of the proposed commitment information gain.

  • The paper compares the first step at which factual and hallucinated answers can be identified. However, it does not separately consider the effect of answer length, tokenization, or the position of tokens that determine the answer. These factors can change this first step even if factual correctness stays the same. Therefore, it is not clear if earlier commitment is related to factual correctness or mainly comes from the decoding order.

Questions for authors
  • Can the authors clarify why Equation 4 compares different masked sets, while Equation 2 defines information gain over the same remaining positions? It would be helpful to compute TI using the same unresolved positions before and after each commitment.

  • Can the authors compare TI with random step selection and selection based only on commitment confidence or masked-set size? This would help show if Informativeness captures information beyond the remasking policy.

  • Can the authors clarify what a commitment event is in DiffusionGemma? It is not clear how entropy at fixed answer positions has the same meaning as the original TI definition.

  • Can the authors show if Early Factual Commitment still holds when factual and hallucinated responses have the same length and answer-token position?

5 comments
AuthorsOpenReview ↗

Thank you for your careful assessment and feedback!

We appreciate the reviewer’s careful assessment and constructive feedback. Below, we address the questions raised and provide additional clarification.

W1 Q1.

Thank you for raising these concerns. We acknowledge that Eq. (4) includes a contribution from changes in the masked-position set, in addition to changes in uncertainty at the remaining positions. We have included additional results below. Please see the “On Eq. (4) versus Eq. (2)” section of our top-level response, where we explain this distinction and report an ablation that computes entropy changes over the same positions at both endpoints.

DatasetModelEq. (4)Eq. (2)Δ
TriviaQALLaDA85.6285.41−0.21
TriviaQADream82.3479.83−2.51
TriviaQAGemma87.0487.18+0.14
HotpotQALLaDA83.8183.97+0.15
HotpotQADream78.8677.63−1.23
HotpotQAGemma82.6382.78+0.15
CommonsenseQALLaDA81.7681.43−0.34
CommonsenseQADream78.8778.33−0.54
CommonsenseQAGemma74.7074.74+0.04

W2 Q2.

Thank you for raising the thoughtful question! Below we include results of Ours-TI against various step-selection methods, including random selection, low / high commit confidence, and low / high masked-set size, to choose 5% (7 steps) out of the total 128 steps. We vary only the step-selection rule, keeping the candidate steps, remasking strategy, final-step component, and the calibration set fixed. We also include mean calibrated weights of the averaged steps, wavgw_\text{avg}, and final step wfinalw_\text{final}.

| Dataset | Step-selection rule | �0��0� | AUROC (%) | Δ vs. TI | | :--- | :--- | ---: | ---: | ---: | ---: | | TriviaQA | TI: informativeness | 0.68 | 0.32 | 85.62 | — | | - | Random (20 seeds) | 0.23 | 0.77 | 77.93 | −7.68 | | - | Highest commitment confidence | 0.18 | 0.82 | 78.98 | −6.64 | | - | Lowest commitment confidence | 0.65 | 0.35 | 85.11 | −0.51 | | - | Largest masked-set size | 0.19 | 0.81 | 78.70 | −6.92 | | - | Smallest masked-set size (LME) | 1.00 | 0.00 | 85.97 | +0.36 | | HotpotQA | TI: informativeness | 0.50 | 0.50 | 83.81 | — | | - | Random (20 seeds) | 0.18 | 0.81 | 75.79 | −8.02 | | - | Highest commitment confidence | 0.10 | 0.90 | 73.95 | −9.86 | | - | Lowest commitment confidence | 0.49 | 0.51 | 83.13 | −0.68 | | - | Largest masked-set size | 0.01 | 0.99 | 72.65 | −11.16 | | - | Smallest masked-set size (LME) | 0.51 | 0.49 | 84.27 | +0.46 |

TI improves over random selection by 7.68 and 8.02 AUROC points on TriviaQA and HotpotQA, respectively. It also outperforms selection by highest commitment confidence and largest masked-set size.

The comparisons with lowest commitment confidence and smallest masked-set size are closer. This is consistent with the motivation behind LME under confidence-based remasking: higher-confidence positions are preferentially committed, leaving a progressively smaller set of unresolved positions. Their competitive performance may therefore reflect Early Factual Commitment and uncertainty signals related to those that make LME effective in this setting. We have updated our manuscript to include this result.

W3 Q3.

Our DiffusionGemma adaptation uses a pristine/modified partition of the canvas. Because the canvas is initialized with random tokens rather than the MASK token, we track each position’s modification history. Positions that have never been modified belong to the pristine region, which serves as an operational counterpart of the masked region in our entropy computation. Positions that have been modified at least once belong to the generated region. This is similar to our settings for LLaDA-2.1-mini.

Accordingly, the event corresponding to commitment in this adaptation is a position’s first recorded modification (or when a position’s predictive entropy first falls below the entropy bound of 10−310^{−3}, which differs from the irreversible mask-to-token commitment assumed in the original formulation. We have updated these definitions and implementation details in our manuscript.

W4 Q4.

We thank the reviewer again for your insightful question! Empirically, we observe that answer-bearing tokens in factual responses tend to become confident earlier in the trajectory, allowing the factual answer to become identifiable before the surrounding response is fully generated. In hallucinated responses, uncertainty about the key content persist, delaying its commitment.

As a result, this mechanism could produce different commitment times for answer-bearing tokens even when responses have the same length and answer-token position. Others, (such as [2]), have reported early stabilization of correct answer predictions and continued late-stage changes in incorrect GSM8K predictions.

[2] Li et al., "Diffusion Language Models Know the Answer Before Decoding".

I thank the authors for the rebuttal. My concerns have been partly addressed. The fixed-position ablation is helpful to clarify the composition effect in Eq. (4). However, I believe the basis for the theoretical derivation is still not technically sound, information gain and the composition term should be modified and more carefully designed.

The new step-selection ablation is also helpful. However, the smallest masked-set selection slightly outperforms TI on both datasets, while lowest-confidence selection achieves comparable results. The gain of informativeness-based selection is not distinctively shown, especially since the combination weights are recalibrated for each strategy.The definition is still inconsistent between commitment based on first modification and entropy threshold crossing. And the response to Early Factual Commitment does not provide the requested controlled analysis of answer length, tokenization, and answer-token position, or any of that ablation, which can be useful to seperate confounding effects.

Overall, I agree the proposed approach is simple and shows promising empirical performance. However, I think the issues of theoretical interpretation, performance gain, and consistency across architectures require further clarification.

We thank the reviewer for the prompt follow-up and thoughtful comments on interpretability and empirical performance. We view the simplicity of our approach as an important strength: it extracts uncertainty signals from a single generation trajectory without training an additional neural detector. This also motivates future work on using intermediate uncertainty signals to support decisions during generation. We address the follow-up concerns below.

On the calibration set and combination weights. All step-selection strategies use the same calibration set. In our original ablation, the combination weights were calibrated separately for each strategy, allowing each detector variant to combine its selected-step entropy with the final-step component under the same calibration-data budget.

To address the reviewer’s concern about the influence of weight recalibration, we additionally evaluate the selectors using the same frozen combination weights under confidence-based decoding. The results below show that TI’s advantages over alternatives remain observable without strategy-specific weight fitting.

Step-selection strategyTriviaQA AUROC (%)HotpotQA AUROC (%)
TI: informativeness85.4884.50
Random74.7577.60
Highest commitment confidence70.1870.56
Lowest commitment confidence84.7583.67
Largest masked-set size68.8469.86
Smallest masked-set size86.2884.65

On the effectiveness of LME, (smallest-masked-set selection), and lowest-confidence selection. We acknowledge that smallest-masked-set selection achieves the highest point estimates under confidence-based decoding, while lowest-confidence selection also performs competitively. These findings are consistent with the motivation for LME: selecting the smallest masked sets targets late decoding steps, where confidence-based decoding tends to concentrate unresolved uncertainty.

To examine how these comparisons depend on the generation policy, we also evaluate the selectors under random remasking with a (10% steps budget):

Step-selection strategyTriviaQA AUROC (%)HotpotQA AUROC (%)
TI: informativeness73.3076.08
Random70.6072.42
Highest commitment confidence68.0375.11
Lowest commitment confidence66.6465.02
Largest masked-set size64.3363.02
Smallest masked-set size50.2860.16

Smallest-masked-set and lowest-confidence selection perform substantially worse in this setting. These results support a more qualified interpretation of our contribution: late-step selection is particularly effective under confidence-based decoding, while selecting steps using observed entropy changes provides an effective alternative when the generation policy changes. We will revise the manuscript to make this dependence explicit.

On Early Factual Commitment. As requested by the reviewer, we conducted a controlled study of Early Factual Commitment using LLaDA-8B-Instruct with 128 decoding steps under confidence-based remasking. We compare correct and hallucinated responses only within strata sharing the exact decoded-answer length in model tokens, (L), and the exact answer-token position, (p). This holds token count and answer-token position fixed within each comparison. We retain strata containing at least eight correct and eight hallucinated responses. The four eligible strata in each dataset correspond to p=0p=0 and (L∈1,2,3,4)(L\in{1,2,3,4}). We report the matched difference in mean commitment step, (Δtc=tc,hallucinated−tc,correct)(\Delta t_c=t_{c,\mathrm{hallucinated}}-t_{c,\mathrm{correct}}), where a positive value indicates earlier commitment for correct answers.

DatasetRetained examplesMatched Δtc\Delta t_c in steps (95% CI)Matched AUROC, % (95% CI)
TriviaQA707+1.41 [+0.70, +2.32]69.90 [65.14, 74.47]
HotpotQA642+1.83 [+0.97, +3.05]71.98 [66.43, 77.69]

After controlling for these factors, correct answers commit earlier by an average of 1.41 step on TriviaQA and 1.83 steps on HotpotQA in the matched analysis. These results support the presence of Early Factual Commitment within the evaluated strata after controlling for answer-token count and position.

Reviewer_Q2qvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates hallucination detection in dLLMs by analyzing uncertainty during the denoising process. The authors observe that factual answers tend to become identifiable earlier than hallucinated ones. They propose two detection methods: Late Mask Entropy (LME), and Token Informativeness (TI).

The methods operate on a single generation without training a separate detector architecture, although TI requires labeled calibration data. Experiments across LLaDA, Dream, and DiffusionGemma show competitive AUROC against several training-free and supervised baselines. Additional experiments investigate remasking strategies and sensitivity to training-data availability.

Critical strengths

Lightweight hallucination detection - The proposed approach extracts useful uncertainty signals from a single denoising trajectory without requiring a separately trained detector.

Reasonable experimental validation - The evaluation includes three dLLMs, multiple QA benchmarks, and different decoding configurations. The results show that LME is effective on LLaDA and Dream, while TI remains more robust when late-stage entropy becomes unreliable. The random-remasking experiments provide additional support for adaptive commitment-step selection.

Critical weaknesses

1. Limited novelty and insufficient comparison with closely related prior work

A major concern is the absence of OSCAR (Shah et al., 2026; arXiv:2604.01624), which already investigates hallucination detection using diffusion-model commitment uncertainty, entropy-based signals, and the temporal formation of hallucinations without training a specialized classifier.

Although the present work uses single-trajectory masked entropy rather than OSCAR's cross-chain entropy, the underlying motivation substantially overlaps. The proposed contribution appears to be primarily a simpler uncertainty aggregation strategy rather than a fundamentally new framework for hallucination detection.

2. Unclear interpretation of Token Informativeness and calibration fairness.

The definition of Token Informativeness is unclear. Equation (2) compares uncertainty over the same token positions, while Equation (4) uses different masked-token sets. This makes it difficult to tell whether the entropy change reflects meaningful information gain or simply the token removal. Also, TI uses 200 calibration examples, even in experiments with only 150 training samples. It is unclear how this additional labeled data is accounted for in the comparisons.

AI assistance and incomplete editing

Page 1, line 43 - [CITE] Page 3, line 138 - [PLEASE CITE SEPARATELY]

Questions for authors

Novelty: How does this method differ from OSCAR (Shah et al., 2026)? Can the authors compare detection performance and computational cost under the same settings?

Information Gain: Why does Equation (4) use changing masked-token sets instead of the fixed positions in Equation (2)? Would this affect detection performance?

Calibration: How is the 200-sample calibration set handled in experiments using only 150 training examples? Is the comparison fair across methods?

6 comments
AuthorsOpenReview ↗

Thank you for your thoughtful review and suggestions!

We thank the reviewer for the thoughtful review and constructive suggestions. We provide our responses and clarifications below.

W1, Q1. Novelty and comparison with OSCAR [1].

We thank the reviewer for highlighting this relevant work. Although both approaches study uncertainty during diffusion-based generation, our method differs from OSCAR in two key respects:

  1. Uncertainty signal and sampling requirements. OSCAR measures cross-path entropy by constructing an empirical distribution of token identities at each aligned position across multiple denoising trajectories with different reveal orders. It therefore captures disagreement across trajectories. In contrast, LME and TI compute predictive entropy from the model’s vocabulary distributions along a single denoising trajectory. LME aggregates late-stage entropy, while TI uses temporal changes in region-averaged entropy to select informative steps and combines their entropy with a final-step feature.
  2. Detection objective and use of uncertainty. OSCAR uses cross-path uncertainty to localize uncertain token positions and support targeted remasking and refinement. Our method focuses on response-level hallucination detection, extracting uncertainty features from an existing generation without requiring additional trajectories or refinement. Regarding supervision, LME requires no labeled calibration data, whereas TI uses labeled examples to calibrate its feature combination. OSCAR’s core cross-path entropy statistic likewise requires no labeled training data.

Our contribution thus centers on extracting useful hallucination-detection signals from a single trajectory, complementing OSCAR’s cross-trajectory approach. We report our reproduction of OSCAR’s results below, as well as in the top-level official comment and have added a discussion of OSCAR to Section 2 of the revised manuscript.

DatasetRepository-selected: accuracy (%)Repository-selected: best AUROC (%)Our question set: accuracy (%)Our question set: best AUROC (%)Ours-TI AUROC (%)
TriviaQA31.067.7226.264.5386.61
HotpotQA32.069.6540.366.5086.56

W2, Q2. Information Gain:

Thank you for raising these concerns. We acknowledge that Eq. (4) includes a contribution from changes in the masked-position set, in addition to changes in uncertainty at the remaining positions. Please see our results below as well as the “On Eq. (4) versus Eq. (2)” section of our top-level response, where we explain this distinction and report an ablation that computes entropy changes over the same positions at both endpoints.

DatasetModelEq. (4)Eq. (2)Δ
TriviaQALLaDA85.6285.41−0.21
TriviaQADream82.3479.83−2.51
TriviaQAGemma87.0487.18+0.14
HotpotQALLaDA83.8183.97+0.15
HotpotQADream78.8677.63−1.23
HotpotQAGemma82.6382.78+0.15
CommonsenseQALLaDA81.7681.43−0.34
CommonsenseQADream78.8778.33−0.54
CommonsenseQAGemma74.7074.74+0.04

W2. Q3. Calibration Set:

Please see “On training, calibration, and test splits” in our top-level response for the sample counts and partitioning protocols. In particular, all main-paper results use a shared training/calibration set and a separate held-out test set; the appendix experiments that use separate training and calibration sets are described explicitly.

AI assistance and incomplete editing

Thank you for pointing out these editing oversights. We have removed the placeholder text and added the appropriate citations in the revised manuscript.

[1] Shah et al., “OSCAR : Orchestrated Self-verification and Cross-path Refinement,” In COLM 2026

Reviewer_Q2qvOpenReview ↗

thanks for the rebuttal - check concerns below

Thank you to the authors for the detailed rebuttal.

1. Novelty and OSCAR comparison

I acknowledge the distinction between single-trajectory predictive entropy and OSCAR's cross-trajectory uncertainty, particularly the computational advantage of avoiding multiple generations. This adds to the strength of the submitted paper.

However, the OSCAR reproduction results remain concerning. The original paper reports AUROC values of 89.7% on TriviaQA and 86.7% on HotpotQA, whereas the reproduced results are substantially lower (67.72% and 69.65%). Could the authors clarify the evaluation protocol and explain this discrepancy? A comparison under matched generation settings, evaluation labels, and computational budgets would strengthen the empirical claims significantly.

2. Token Informativeness

The Eq. (2) versus Eq. (4) ablation largely addresses my original concern, although Eq. (4) still captures changes in masked-set composition alongside predictive uncertainty.

I also noticed that Table 5 shows the simpler smallest-masked-set selection slightly outperforming TI on TriviaQA (85.97 vs. 85.62) and HotpotQA (84.27 vs. 83.81), although the differences are within the reported confidence intervals. This raises questions about the specific advantage of informativeness-based selection over simpler heuristics.

3. Calibration and presentation

The clarification regarding shared training/calibration splits resolves my concern about additional labeled data. Thanks for the correction of the citation placeholders.

The rebuttal addresses several of my concerns, and I recognize the practical value of single-trajectory hallucination detection. However, the unexplained OSCAR reproduction gap and the limited advantage over simpler step-selection heuristics leave my concerns regarding novelty and empirical justification only partially resolved.

I maintain by rating currently.

AuthorsOpenReview ↗

Thank you for your prompt response!

We thank the reviewer for your prompt response and address some of the follow-up concerns below.

On Novelty and OSCAR comparison. Our earlier OSCAR results were unreliable because issues in the released LLaDA generation implementation produced severely degraded, often nonsensical answers in our runs. Specifically, the generation code suppressed end-of-sequence tokens and did not apply the Instruct checkpoint’s chat template. We corrected these generation-handling issues and reran the experiments. We apologize for reporting the earlier comparison before identifying this problem; the updated results below replace those estimates.

Following these fixes, we used OSCAR’s released generation and scoring pipeline on an NVIDIA H200 GPU, with LLaDA-8B-Instruct, 8 generation paths, and using 128 generation steps, matching the paper’s reported backbone, path count, and configured step budget. However, we did use a different evaluation judge, GPT-5.4-nano, instead of the reported GPT-4o due to API availability.

We also found that the reported AUROC depends substantially on the scoring variant. Across our runs, taking the highest AUROC among the repository’s variants raises the reported value by up to 7 points relative to mean cross-path entropy. For our primary comparison, we therefore compute the token-level cross-path entropy (defined in OSCAR’s Eq. (1)). Furthermore, report alternative aggregations results below to make this sensitivity transparent.

We hypothesize that other plausible contributors include the judge model and rubric, question selection, and variation across generation seeds. The evaluation stage also matters: OSCAR distinguishes entropy-only detection from its full pipeline with correction and judge relabeling. Consequently, matching the backbone and path count alone does not ensure that the same responses and correctness targets are being evaluated. For the purpose of this rebuttal, we are unable to attribute the remaining gap to a single factor.

Updated results from OSCAR

Method / scoreGeneration pathsTriviaQA AUROC (%)HotpotQA AUROC (%)
OSCAR: mean cross-path entropy — primary comparison871.1 [66.1, 75.5]70.7 [65.2, 76.5]
OSCAR: top-20% cross-path entropy871.471.6
OSCAR: maximum cross-path entropy873.975.0
Ours-TI — reference results on its own generations185.683.8

The updated OSCAR numbers are closer to the published results, although they are below the paper’s reported 89.7% on TriviaQA and 86.7% on HotpotQA for the 128-step LLaDA judge evaluation. Nevertheless, our methodological distinction remains clear. We will revise the manuscript to include these corrected results.

On Token Informativeness.

We clarify that the smallest-masked-set baseline selects an identical set of steps as LME. Its strong performance in the step-selection ablation is therefore consistent with Table 2, where Ours-LME can outperform TI under confidence-based remasking. In this setting, selecting the smallest masked sets corresponds to selecting the final decoding steps, where late masked-position entropy is particularly effective. However, under random remasking, TI achieves 73.30% and 76.08% AUROC on TriviaQA and HotpotQA, respectively, compared with 50.28% and 60.16% for smallest-masked-set selection. Our DiffusionGemma results similarly demonstrate this limitation. These findings support TI’s greater robustness across the evaluated decoding settings.

Finally, we acknowledge that our Eq. (4) still captures changes in masked-set composition, and therefore propose a revision to our informativeness score to explicitly separate the fixed-position predictive entropy change from the masked-set composition effect:

Let R(i)=M(i+1)R^{(i)}=M^{(i+1)} denote the positions that remain masked and B(i)=M(i)∖R(i)B^{(i)}=M^{(i)}\setminus R^{(i)} the newly committed positions. Let EA(s)E_A^{(s)} denote the average predictive entropy over positions AA, evaluated at step ss. For nested masked sets with a nonempty remaining region, the original score admits the exact decomposition

I(i)=U(i)+Γ(i),I^{(i)} = U^{(i)} + \Gamma^{(i)},

where

U(i)=ER(i)(i)−ER(i)(i+1)U^{(i)} = E_{R^{(i)}}^{(i)} - E_{R^{(i)}}^{(i+1)}

is the fixed-position entropy change in Eq. (2), and

Γ(i)=∣B(i)∣∣M(i)∣(EB(i)(i)−ER(i)(i))\Gamma^{(i)} = \frac{|B^{(i)}|}{|M^{(i)}|} \left( E_{B^{(i)}}^{(i)} - E_{R^{(i)}}^{(i)} \right)

is the composition term.

This revision explicitly states if a contribution already present in the original estimator and preserves TI's selection rule, which ranks steps by ∣U(i)+Γ(i)∣\lvert U^{(i)}+\Gamma^{(i)}\rvert. We will describe Eq. (2) as a realized fixed-position predictive entropy change and Eq. (4) as a change in mean entropy over the evolving masked region.

We will update these definitions in our updated manuscript.

Reviewer_x2QtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to detect hallucinations in diffusion language models from the steps used to generate one answer. Its starting observation is that, in LLaDA, correct answers often reveal key answer information earlier than wrong answers. The authors separate positions that are still masked from positions that already contain generated tokens and track their predictive uncertainty. They propose two scores. Late Mask Entropy averages uncertainty over the last few steps with masked positions. Token Informativeness selects a small set of steps with large changes in masked-region entropy and combines their entropy with a final-step score. The experiments cover several diffusion models, four QA datasets, and different decoding settings. Late Mask Entropy works well on LLaDA and Dream but is weak on DiffusionGemma, while Token Informativeness is more competitive there. The main goal is to match stronger detectors with simple statistics and lightweight calibration.

Critical strengths
  • A simple detector could be easier to use and train than a large extra model.
  • Using one generated answer could save time compared with methods that need several answers.
  • Tests with different decoding rules help show when the method works.
Critical weaknesses
  • The practical benefit is not fully measured. The cost table leaves out answer generation and feature extraction. The paper should measure total time and test whether its score helps a system avoid wrong answers or send doubtful cases for checking.
  • The entropy score compares different sets of masked positions. Its change may partly come from removing positions, even when uncertainty at the remaining positions stays the same. The train, calibration, and test splits also need clearer descriptions.
Questions for authors

refer to weakness

3 comments
AuthorsOpenReview ↗

Thank you for your positive review!

We are glad the reviewer found the paper interesting and thank the reviewer for the positive review! We address some of the concerns and questions below:

W1.

We thank the reviewer for this suggestion. Our current cost table isolates detector fitting from cached features; it does not measure end-to-end inference time. For LME and TI, feature extraction uses the model outputs already produced during a single denoising trajectory and requires no additional answer-generation runs or model forward passes. The incremental computation consists of entropy calculation and aggregation, together with step selection for TI. We therefore expect a small overhead relative to answer generation, but agree that this should be quantified directly.

We will supplement the cost analysis with end-to-end timing that includes answer generation, feature extraction, and detector scoring in the final version of the manuscript. We will also report the incremental overhead relative to generation alone under the same hardware and decoding settings, while keeping the one-time fitting and calibration costs separate.

W2.

Thank you for raising these concerns. We acknowledge that Eq. (4) includes a contribution from changes in the masked-position set, in addition to changes in uncertainty at the remaining positions. Please see our results below as well as the “On Eq. (4) versus Eq. (2)” section of our top-level response, where we explain this distinction and report an ablation that computes entropy changes over the same positions at both endpoints.

DatasetModelEq. (4)Eq. (2)Δ
TriviaQALLaDA85.6285.41−0.21
TriviaQADream82.3479.83−2.51
TriviaQAGemma87.0487.18+0.14
HotpotQALLaDA83.8183.97+0.15
HotpotQADream78.8677.63−1.23
HotpotQAGemma82.6382.78+0.15
CommonsenseQALLaDA81.7681.43−0.34
CommonsenseQADream78.8778.33−0.54
CommonsenseQAGemma74.7074.74+0.04

Please also see “On training, calibration, and test splits” in our top-level response for the sample counts and partitioning protocols. In particular, all main-paper results use a shared training/calibration set and a separate held-out test set. We have included these changes in our revised manuscript.

I thank the authors for their detailed rebuttal. I think my current score already reflects my assessment, so I will keep it unchanged.

Thank you for your positive assessment and thoughtful feedback! We sincerely appreciate your time and consideration of our work.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.