acceptodds
Under review as a conference paper at ICLR 2027

Palimpsest: Bidirectional Scoring for Span-Level Hallucination Detection

Abstract

Large language models can produce "hallucinated" statements that sound fluent but are ungrounded in reality, motivating reliable detection. Even though support for a span of statement can depend on words that follow it, many evaluators score text left to right. Motivated by this shortcoming, we propose Palimpsest, a span-level detector that uses masked diffusion language models (MDLMs), which evaluates the factuality of a span of text using its context from both sides. We compare Palimpsest to autoregressive (AR) models with AUROC and find a) on BUMP with human-annotated spans, 8B MDLM scorers match the strongest tested AR model, which has 3.9 times as many nominal parameters, and b) on LLM-AggreFact they lag behind slightly while being more parameter-efficient. On HalluEntity, LLaDA-8B-Base outperforms the strongest tested AR scorer, a 31B checkpoint. Thus, Palimpsest can rival larger AR-based methods for span-level hallucination detection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hallucination detection, factuality evaluation, masked diffusion language models, autoregressive language models, span-level verification, foundation models, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y7dMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a span-level hallucination detection framework based on masked diffusion language models. It introduces two scoring strategies: Recovery, which measures the reconstruction likelihood of masked spans, and YN, which predicts factual support through a masked yes/no token. Experiments on three benchmarks demonstrate competitive performance against larger autoregressive models, particularly when precise candidate spans are provided.

Critical strengths
  1. The paper explores an interesting application of masked diffusion language models to fine-grained hallucination detection. Leveraging bidirectional context for factuality assessment is well motivated.
  2. The experiments cover multiple datasets, model families, and scoring strategies. The comparison between Dream and Qwen provides useful evidence, while the span proposal experiments reveal important practical limitations.
Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for bidirectionality.
  2. Strong dependence on precise candidate spans.
Questions for authors
  1. How does the proposed method compare with conventional bidirectional masked language models using the same recovery-based scoring strategy? What benefits specifically arise from diffusion-based training?
  2. Could the authors provide a controlled ablation comparing left-only, right-only, and bidirectional conditioning within the same MDLM to verify the contribution of bidirectionality?
  3. Given the substantial performance degradation with automatically proposed spans, how would the method perform in an end-to-end hallucination detection setting without gold span boundaries?
  4. The paper claims improved efficiency based on parameter counts. Could the authors provide actual inference latency, GPU memory consumption, and computational cost comparisons?
2 comments

We thank the reviewer for the careful assessment and for recognizing that using bidirectional context for factuality assessment is well motivated. We also appreciate the reviewer’s recognition that our experiments across datasets, model families, and scoring strategies provide useful evidence through the Dream vs. Qwen comparison and reveal practical limitations through the span-proposal experiments. We address the concerns below.

New ablataion study: the bidirectionality effect

We thank the reviewer for raising the question regarding the effect of the bidirectionality of MDLMs. To illustrate the performance gain from the MDLM’s bidirectionality, we conducted the following new ablation study, in which the reference is always retained and only the visible context within the generated text changes. We designed the following three context conditions:

  1. both: the original X′.
  2. left-only: X′ truncated immediately after the last masked token, so the input is R(X) + the prefix of X + the masks.
  3. right-only: X′ with the prefix before the span removed, so the input is R(X) + the masks + the suffix of X.
  4. left-only-right-mask: X′ where tokens after the last masked token are converted to the mask token as well, so the input is R(X) + the prefix of X + the masks + the suffix of X converted to mask token.
  5. right-only-left-mask: X′ where tokens before the first masked token are converted to the mask token as well, so the input is R(X) + the prefix of X converted to mask token + the masks + the suffix of X.
ModelContext within XHalluEntity AUROCBUMP AUROC
LLaDA-8B-Baseboth0.800.95
LLaDA-8B-Baseleft-only0.600.85
LLaDA-8B-Baseright-only0.580.74
LLaDA-8B-Baseleft-only-right-mask0.610.86
LLaDA-8B-Baseright-only-left-mask0.730.88

Our results show that context on both sides of the span yields clearly better performance than left-only or right-only context.

New ablation study: Comparison with conventional bidirectional MLMs

We thank the reviewer for raising hte question regarding applying our method with conventional bidirectional masked language models. Below, we compare the performance of these models (RoBERTa, ModernBERT, and BERT) with LLaDA-8B:

ModelHalluEntityBUMPLLM-AggreFact
RoBERTa-large0.790.910.47
ModernBERT-large0.790.870.40
BERT-large-uncased0.780.670.43
LLaDA-8B0.800.950.70

These conventional bidirectional MLMs achieve performance comparable to that of LLaDA-8B only on HalluEntity. We think this is related to the length of the candidate spans. As candidate spans grow longer, the performance of conventional bidirectional MLMs drops significantly. We believe this is mainly because these models are trained to recover only a fixed percentage of masked tokens. Sicne LLM-AggreFact involves the longest span, conventional MLMs is expected to perform worse.

New results: Inference latency, GPU memory, and compute

We thank the reviewer for asking for more computational performance results. As the paper already establishes, Palimpsest is parameter efficient: its 7–8B scorers match the strongest AR judge, gemma-4-31B-it, which has 3.9 times as many parameters. To show how this translates into inference cost, we conducted a new study and measured latency, peak GPU memory usage, and FLOPs on a single H100 GPU.

ModelMedian latency per span (ms)Peak GPU memory usage (GB)FLOPs per span (TFLOPs)
LLaDA-8B-Base4516.614.5
gemma-4-31B19166.664.7

These measurements are consistent with Palimpsest’s parameter efficiency.

Methodological novelty

To the best of our knowledge, our work is the first to use MDLMs as verifiers: Palimpsest uses the bidirectional context of an MDLM to build a family of span-level scores that can verify text from any source, with or without a reference. In contrast, prior work on hallucination in diffusion LMs [1, 2] analyzes the diffusion model's own generation trace, with the goal of reducing hallucination in what that model itself generates. That goal is different, and those methods cannot verify text that another model produced. In addition, we compare MDLM verifiers systematically with AR judges, and our new experiments above show (i) that the gain comes from bidirectional context and (ii) that conventional bidirectional MLMs do not give the same result on longer spans.

[1] Tracedet: Hallucination detection from the decoding trace of diffusion large language models [2] TDGNet: Hallucination Detection in Diffusion Language Models via Temporal Dynamic Graphs

On candidate span selection

Palimpsest is a verification method that operates on candidate spans. This requirement is not specific to Palimpsest. Span verifiers as a class, including the AR-based methods that we compare against and the methods evaluated in the original HalluEntity study, all score a given span. Accordingly, our experiments follow the existing benchmarks: we use the gold spans that are part of HalluEntity, as the original HalluEntity study does, and the full sentence on LLM-AggreFact. The contribution of Palimpsest is the use of MDLMs and their bidirectional context for verification, and this is the focus of our study. How to propose better spans is an interesting research question of its own. It is beyond the scope of this work, and any improvement in span proposal directly benefits Palimpsest, because Palimpsest can verify candidates from any proposer.

End-to-end performance

Our existing experiments already illustrate the end-to-end performance of our method. Figure 5 in the paper shows the AUROC on BUMP as the candidate spans change from human-annotated spans to spans proposed by a language model or entire sentences. As Figure 5 shows, replacing BUMP’s human-annotated spans with coarser spaCy sentence spans or spans proposed by SmolLM3-3B, which are less precise than the human-annotated spans, sharply reduces the AUROC of nearly all Palimpsest methods.

We hope this response resolves your concerns and that you will consider raising your score. We are happy to answer any further questions.

Reviewer_fGYmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PALIMPSEST, a span-level hallucination detection method based on masked diffusion language models. The main idea is to use bidirectional contextual information to assess whether a candidate span is factually supported. The paper introduce two approaches: PALIMPSEST-Recovery (S1/S2), which measures the likelihood of reconstructing masked spans, and PALIMPSEST-YN (F1/F2), which estimates whether a candidate span is supported through a masked yes/no prediction. The methods are evaluated against autoregressive (AR) language models and existing hallucination detectors on HalluEntity, BUMP, and LLM-AggreFact. The results suggest that 7B–8B diffusion language models can achieve competitive detection performance.

Critical strengths
  • The paper explores the use of masked diffusion language models as external hallucination scorers.

  • The distinction between single-pass reconstruction (S1) and sequential reconstruction (S2) is interesting, and the experiments provide insight into their different behaviors.

  • The reference-free HalluEntity results are promising, with recovery-based scoring achieving approximately 0.80 AUROC.

  • The authors evaluate several datasets and baselines, provide bootstrap variability estimates, and explicitly investigate sensitivity to span proposal.

Critical weaknesses
  • The paper motivates PALIMPSEST as a span-level hallucination detection method, but it relies on predefined candidate spans rather than identifying hallucinated spans directly. This limits its applicability to end-to-end hallucination detection.

  • On BUMP, the strong results depend heavily on human-annotated span boundaries. Performance degrades substantially when automatically proposed or sentence-level spans are used. Also, smaller dedicated hallucination detectors outperform PALIMPSEST in these settings.

  • The performance improvements are not consistently substantial compared with strong AR baselines. PALIMPSEST matches the strongest AR baseline on BUMP but performs worse on LLM-AggreFact.

  • The comparison intended to isolate the benefits of bidirectionality is not fully controlled, as the MDLM and AR baselines use different scoring formulations and prompts. It is unclear how much of the improvement can be attributed specifically to bidirectional conditioning.

  • Several AR and MDLM-based scoring configurations obtain substantially below-chance AUROC on HalluEntity. These unusual results require further investigation.

  • The paper does not analyze how hallucination detection accuracy varies with candidate span length. Appendix C reports decreasing correlation between S1 and S2 as span length increases, this does not establish which method performs better on longer spans.

  • Equation F appears inconsistent. z_yes and z_no are defined as probabilities but apply exponential normalization.

Questions for authors
  • Can the authors also report AUPRC and precision-based measures, particularly under class imbalance?
1 comment

We thank the reviewer for the detailed review and for highlighting the S1 vs. S2 distinction and our bootstrap and span-proposal analyses. We address each point below.

New ablataion study: the bidirectionality effect

We thank the reviewer for raising the question regarding the effect of the bidirectionality of MDLMs. To illustrate the performance gain from the MDLM’s bidirectionality, we conducted the following new ablation study, in which the reference is always retained and only the visible context within the generated text changes. We designed the following three context conditions:

  1. both: the original X′.
  2. left-only: X′ truncated immediately after the last masked token, so the input is R(X) + the prefix of X + the masks.
  3. right-only: X′ with the prefix before the span removed, so the input is R(X) + the masks + the suffix of X.
  4. left-only-right-mask: X′ where tokens after the last masked token are converted to the mask token as well, so the input is R(X) + the prefix of X + the masks + the suffix of X converted to mask token.
  5. right-only-left-mask: X′ where tokens before the first masked token are converted to the mask token as well, so the input is R(X) + the prefix of X converted to mask token + the masks + the suffix of X.
ModelContext within XHalluEntity AUROCBUMP AUROC
LLaDA-8B-Baseboth0.800.95
LLaDA-8B-Baseleft-only0.600.85
LLaDA-8B-Baseright-only0.580.74
LLaDA-8B-Baseleft-only-right-mask0.610.86
LLaDA-8B-Baseright-only-left-mask0.730.88

Our results show that context on both sides of the span yields clearly better performance than left-only or right-only context.

New AUPRC table

We thank the reviewer for point our the need for AUPRC results. Below is the new AUPRC table:

DatasetLLaDA-8Bgemma-4-31B
HalluEntity0.32 ± 0.040.37 ± 0.07
BUMP0.95 ± 0.010.96 ± 0.01
LLM-AggreFact0.70 ± 0.010.66 ± 0.01

Our results show that LLaDA-8B and gemma-4-31B achieve broadly comparable performance in terms of AUPRC, as their confidence intervals overlap (except on LLM-AggreFact, where LLaDA-8B performs better). LLaDA-8B remains competitive despite its smaller size.

On candidate span selection

Palimpsest is a verification method that operates on candidate spans. This requirement is not specific to Palimpsest. Span verifiers as a class, including the AR-based methods that we compare against and the methods evaluated in the original HalluEntity study, all score a given span. Accordingly, our experiments follow the existing benchmarks: we use the gold spans that are part of HalluEntity, as the original HalluEntity study does, and the full sentence on LLM-AggreFact. The contribution of Palimpsest is the use of MDLMs and their bidirectional context for verification, and this is the focus of our study. How to propose better spans is an interesting research question of its own. It is beyond the scope of this work, and any improvement in span proposal directly benefits Palimpsest, because Palimpsest can verify candidates from any proposer.

Gains over strong AR baselines are not consistent

Palimpsest performs worse on LLM-AggreFact because the candidate span used in that dataset is an entire sentence. We hypothesize that better span-proposal algorithms could improve Palimpsest’s performance on LLM-AggreFact.

AUROC below 0.5 on HalluEntity

We re-checked the scoring direction, the label mapping, and the evaluation code for all of these values, and we confirm that there is no error. AUROC can take any value between 0 and 1, and these values show that these AR scorers do not perform well on HalluEntity specifically. We believe one reason for these models’ low performance on HalluEntity is that its claims contain many punctuation marks and function words. MDLMs can recover these tokens more easily than AR models, because MDLMs condition on the text both before and after the masked span.

Accuracy vs. candidate span length

Our existing experiments illustrate the relationship between accuracy and candidate span length. Figure 5 in the paper shows the AUROC on BUMP as the candidate spans change from human-annotated spans to spans proposed by a language model or entire sentences, moving from the most precise to the coarsest spans. As Figure 5 shows, replacing BUMP’s human-annotated spans with coarser spaCy sentence spans or spans proposed by SmolLM3-3B, which are less precise than the human-annotated spans, sharply reduces the AUROC of nearly all Palimpsest methods.

Equation F

Could the reviewer kindly clarify what Eq. F is inconsistent with?

We hope this response resolves your concerns and that you will consider raising your score. We are happy to answer any further questions.

Reviewer_iH8GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PALIMPSEST, a span-level hallucination detection framework based on masked diffusion language models (MDLMs). It introduces two scoring approaches: RECOVERY, which measures how well masked spans can be reconstructed from their context, and YN, which predicts a masked yes/no answer to assess factual support. Decision thresholds are calibrated using split conformal prediction. Experiments on HalluEntity, BUMP, and LLM-AggreFact show that 7–8B MDLM scorers can outperform, match, or slightly trail substantially larger AR scorers, depending on the dataset. The paper also investigates matched-backbone comparisons, candidate span selection, and fine-tuning.

Critical strengths

S1. A simple, training-free use of MDLMs. Masking candidate spans and measuring their recovery likelihood is intuitive and requires no task-specific training.

S2. Transparent experimental reporting. The paper reports bootstrap standard deviations, TPR at fixed FPR budgets, and detailed prompt templates. It also includes experiments with automatically proposed spans, which reveal an important limitation of the method.

Critical weaknesses

W1. Performance depends heavily on gold spans. On BUMP, LLaDA-8B S2 drops from 0.95 AUROC with human spans to 0.72/0.69 with SmolLM3/spaCy spans, while gemma-4-31B R1 remains at 0.92/0.90. Since gold spans are generally unavailable in deployment, this limits the claimed competitiveness.

W2. Most AR baselines perform unexpectedly below chance. On HalluEntity, seven of eight R1/R2 configurations achieve only 0.33–0.49 AUROC, and Muse-Glimmer-30B R1 has zero TPR across all reported FPR budgets (Table 1). Without validating scoring direction and model configurations, the comparison may overstate PALIMPSEST's advantage.

W3. The matched-pair experiment does not isolate bidirectionality. Although Dream-v0-7B and Qwen2.5-7B share a related initialization, they differ in subsequent training objectives, while PALIMPSEST and AR baselines use different scoring mechanisms. The performance differences therefore cannot be attributed solely to bidirectional conditioning.

Questions for authors

Q1. Could the authors address the span-selection robustness issue raised in W1?

Q2. Could the authors clarify the baseline reliability and bidirectionality concerns raised in W2–W3?

2 comments

We thank the reviewer for the constructive feedback, for recognizing Palimpsest as a simple, training-free use of MDLMs, and for acknowledging our transparent experimental reporting. We address the concerns below.

New ablataion study: the bidirectionality effect

We thank the reviewer for raising the question regarding the effect of the bidirectionality of MDLMs. To illustrate the performance gain from the MDLM’s bidirectionality, we conducted the following new ablation study, in which the reference is always retained and only the visible context within the generated text changes. We designed the following three context conditions:

  1. both: the original X′.
  2. left-only: X′ truncated immediately after the last masked token, so the input is R(X) + the prefix of X + the masks.
  3. right-only: X′ with the prefix before the span removed, so the input is R(X) + the masks + the suffix of X.
  4. left-only-right-mask: X′ where tokens after the last masked token are converted to the mask token as well, so the input is R(X) + the prefix of X + the masks + the suffix of X converted to mask token.
  5. right-only-left-mask: X′ where tokens before the first masked token are converted to the mask token as well, so the input is R(X) + the prefix of X converted to mask token + the masks + the suffix of X.
ModelContext within XHalluEntity AUROCBUMP AUROC
LLaDA-8B-Baseboth0.800.95
LLaDA-8B-Baseleft-only0.600.85
LLaDA-8B-Baseright-only0.580.74
LLaDA-8B-Baseleft-only-right-mask0.610.86
LLaDA-8B-Baseright-only-left-mask0.730.88

Our results show that context on both sides of the span yields clearly better performance than left-only or right-only context.

AR baselines with AUROC below 0.5

Thank you for raising this point. We re-checked the scoring direction, the label mapping, and the evaluation code for all of these values, and we confirm that there is no error. AUROC can take any value between 0 and 1, and these values show that these AR scorers do not perform well on HalluEntity specifically. We believe one reason for these models’ low performance on HalluEntity is that its claims contain many punctuation marks and function words. MDLMs can recover these tokens more easily than AR models, because MDLMs condition on the text both before and after the masked span.

On candidate span selection

Palimpsest is a verification method that operates on candidate spans. This requirement is not specific to Palimpsest. Span verifiers as a class, including the AR-based methods that we compare against and the methods evaluated in the original HalluEntity study, all score a given span. Accordingly, our experiments follow the existing benchmarks: we use the gold spans that are part of HalluEntity, as the original HalluEntity study does, and the full sentence on LLM-AggreFact. The contribution of Palimpsest is the use of MDLMs and their bidirectional context for verification, and this is the focus of our study. How to propose better spans is an interesting research question of its own. It is beyond the scope of this work, and any improvement in span proposal directly benefits Palimpsest, because Palimpsest can verify candidates from any proposer.

We hope our response resolves your concerns and that you will consider raising your score. We are happy to answer any further questions.

I appreciate the additional ablation study. However, my concerns about gold span reliance and AR baseline reliability remain. Thus, I maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive reviews.

Summary of the paper. Palimpsest is a training-free method that uses masked diffusion language models (MDLMs) as external, span-level verifiers of generated text. Palimpsest masks a candidate span and scores how well the MDLM recovers it from the context on both sides of the span, with or without a reference, using single-pass (S1) or sequential (S2) recovery. With 7–8B scorers, Palimpsest matches the strongest autoregressive (AR) judge, gemma-4-31B-it, which has 3.9 times as many parameters.

Strengths recognized by the reviewers.

  • A new use of MDLMs (y7dM, fGYm). MDLMs serve as external, fine-grained hallucination scorers, and bidirectional context is "well motivated" for factuality assessment.
  • A simple method (iH8G). Palimpsest is simple, intuitive, and training-free.
  • Useful analysis (fGYm, y7dM). The reviewers found the distinction between single-pass (S1) and sequential (S2) recovery interesting and the Dream vs. Qwen comparison useful.
  • A broad and transparent evaluation (y7dM, fGYm, iH8G). The evaluation covers multiple datasets, model families, scoring strategies, and baselines, and reports bootstrap variability estimates, TPR at fixed FPR, detailed prompt templates, and span-proposal experiments.

New experiments. We thank the reviewers for their questions and suggestions, which have helped us strengthen the paper. In response, we conducted the following additional experiments. The full tables are in the individual responses.

  • Controlled bidirectionality ablation. We keep the reference fixed and change only the context that is visible within the generated text. Context on both sides of the span gives 0.80 AUROC on HalluEntity and 0.95 on BUMP. Every one-sided condition is lower (0.58–0.73 on HalluEntity and 0.74–0.88 on BUMP). This isolates the gain that comes from bidirectional context.
  • Comparison with conventional bidirectional MLMs. RoBERTa, ModernBERT, and BERT are close to Palimpsest on HalluEntity, where the spans are short (0.78–0.79 vs. 0.80). They fall behind as the spans become longer (0.40–0.47 vs. 0.70 on LLM-AggreFact), because they are trained with a fixed, low mask rate.
  • AUPRC results. LLaDA-8B is comparable to gemma-4-31B on HalluEntity and BUMP, and better on LLM-AggreFact.
  • Inference cost. On a single H100 GPU, Palimpsest with LLaDA-8B has about 4.2× lower latency, 4.0× lower peak GPU memory usage, and 4.5× fewer FLOPs per span than gemma-4-31B.

On candidate spans. As discussed in the limitations section of our paper and as noted by the reviewers, Palimpsest is a verification method that operates on candidate spans. This requirement is not specific to Palimpsest. Span verifiers as a class, including the AR-based methods that we compare against and the methods evaluated in the original HalluEntity study, all score a given span. Accordingly, our experiments follow the existing benchmarks: we use the gold spans that are part of HalluEntity, as the original HalluEntity study does, and the full sentence on LLM-AggreFact. The contribution of Palimpsest is the use of MDLMs and their bidirectional context for verification, and this is the focus of our study. How to propose better spans is an interesting research question of its own. It is beyond the scope of this work, and any improvement in span proposal directly benefits Palimpsest, because Palimpsest can verify candidates from any proposer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.