- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a span-level hallucination detection framework based on masked diffusion language models. It introduces two scoring strategies: Recovery, which measures the reconstruction likelihood of masked spans, and YN, which predicts factual support through a masked yes/no token. Experiments on three benchmarks demonstrate competitive performance against larger autoregressive models, particularly when precise candidate spans are provided.
Critical strengths
- The paper explores an interesting application of masked diffusion language models to fine-grained hallucination detection. Leveraging bidirectional context for factuality assessment is well motivated.
- The experiments cover multiple datasets, model families, and scoring strategies. The comparison between Dream and Qwen provides useful evidence, while the span proposal experiments reveal important practical limitations.
Critical weaknesses
- Limited methodological novelty and insufficient evidence for bidirectionality.
- Strong dependence on precise candidate spans.
Questions for authors
- How does the proposed method compare with conventional bidirectional masked language models using the same recovery-based scoring strategy? What benefits specifically arise from diffusion-based training?
- Could the authors provide a controlled ablation comparing left-only, right-only, and bidirectional conditioning within the same MDLM to verify the contribution of bidirectionality?
- Given the substantial performance degradation with automatically proposed spans, how would the method perform in an end-to-end hallucination detection setting without gold span boundaries?
- The paper claims improved efficiency based on parameter counts. Could the authors provide actual inference latency, GPU memory consumption, and computational cost comparisons?
2 comments
Methodological novelty
To the best of our knowledge, our work is the first to use MDLMs as verifiers: Palimpsest uses the bidirectional context of an MDLM to build a family of span-level scores that can verify text from any source, with or without a reference. In contrast, prior work on hallucination in diffusion LMs [1, 2] analyzes the diffusion model's own generation trace, with the goal of reducing hallucination in what that model itself generates. That goal is different, and those methods cannot verify text that another model produced. In addition, we compare MDLM verifiers systematically with AR judges, and our new experiments above show (i) that the gain comes from bidirectional context and (ii) that conventional bidirectional MLMs do not give the same result on longer spans.
[1] Tracedet: Hallucination detection from the decoding trace of diffusion large language models [2] TDGNet: Hallucination Detection in Diffusion Language Models via Temporal Dynamic Graphs
On candidate span selection
Palimpsest is a verification method that operates on candidate spans. This requirement is not specific to Palimpsest. Span verifiers as a class, including the AR-based methods that we compare against and the methods evaluated in the original HalluEntity study, all score a given span. Accordingly, our experiments follow the existing benchmarks: we use the gold spans that are part of HalluEntity, as the original HalluEntity study does, and the full sentence on LLM-AggreFact. The contribution of Palimpsest is the use of MDLMs and their bidirectional context for verification, and this is the focus of our study. How to propose better spans is an interesting research question of its own. It is beyond the scope of this work, and any improvement in span proposal directly benefits Palimpsest, because Palimpsest can verify candidates from any proposer.
End-to-end performance
Our existing experiments already illustrate the end-to-end performance of our method. Figure 5 in the paper shows the AUROC on BUMP as the candidate spans change from human-annotated spans to spans proposed by a language model or entire sentences. As Figure 5 shows, replacing BUMP’s human-annotated spans with coarser spaCy sentence spans or spans proposed by SmolLM3-3B, which are less precise than the human-annotated spans, sharply reduces the AUROC of nearly all Palimpsest methods.
We hope this response resolves your concerns and that you will consider raising your score. We are happy to answer any further questions.
We thank the reviewer for the careful assessment and for recognizing that using bidirectional context for factuality assessment is well motivated. We also appreciate the reviewer’s recognition that our experiments across datasets, model families, and scoring strategies provide useful evidence through the Dream vs. Qwen comparison and reveal practical limitations through the span-proposal experiments. We address the concerns below.
New ablataion study: the bidirectionality effect
We thank the reviewer for raising the question regarding the effect of the bidirectionality of MDLMs. To illustrate the performance gain from the MDLM’s bidirectionality, we conducted the following new ablation study, in which the reference is always retained and only the visible context within the generated text changes. We designed the following three context conditions:
both: the original X′.left-only: X′ truncated immediately after the last masked token, so the input is R(X) + the prefix of X + the masks.right-only: X′ with the prefix before the span removed, so the input is R(X) + the masks + the suffix of X.left-only-right-mask: X′ where tokens after the last masked token are converted to the mask token as well, so the input is R(X) + the prefix of X + the masks + the suffix of X converted to mask token.right-only-left-mask: X′ where tokens before the first masked token are converted to the mask token as well, so the input is R(X) + the prefix of X converted to mask token + the masks + the suffix of X.Our results show that context on both sides of the span yields clearly better performance than left-only or right-only context.
New ablation study: Comparison with conventional bidirectional MLMs
We thank the reviewer for raising hte question regarding applying our method with conventional bidirectional masked language models. Below, we compare the performance of these models (RoBERTa, ModernBERT, and BERT) with LLaDA-8B:
These conventional bidirectional MLMs achieve performance comparable to that of LLaDA-8B only on HalluEntity. We think this is related to the length of the candidate spans. As candidate spans grow longer, the performance of conventional bidirectional MLMs drops significantly. We believe this is mainly because these models are trained to recover only a fixed percentage of masked tokens. Sicne LLM-AggreFact involves the longest span, conventional MLMs is expected to perform worse.
New results: Inference latency, GPU memory, and compute
We thank the reviewer for asking for more computational performance results. As the paper already establishes, Palimpsest is parameter efficient: its 7–8B scorers match the strongest AR judge, gemma-4-31B-it, which has 3.9 times as many parameters. To show how this translates into inference cost, we conducted a new study and measured latency, peak GPU memory usage, and FLOPs on a single H100 GPU.
These measurements are consistent with Palimpsest’s parameter efficiency.