acceptodds
Under review as a conference paper at ICLR 2027

DLIPS: Detecting Hallucinations in Large Language Models via Temporal Lipschitz Dynamics

Abstract

Hallucinations undermine the reliability of large language models (LLMs). Existing hallucination detectors often rely on answer representations, token-level scores, or agreement across sampled responses, but such result-centric signals can obscure transient computational anomalies that emerge during generation and are no longer apparent in the final answer state. We therefore investigate whether hallucinations can be identified from abnormal dynamics in the model's local computational sensitivity, rather than solely from static or aggregated evidence after generation. We propose DLIPS (Dynamic LIPSchitz-based hallucination detection), a process-centric framework that uses temporal Lipschitz dynamics as a signal for response-level hallucination detection. DLIPS tracks local perturbation amplification across prefixes of a single candidate answer by estimating decoder-block Jacobian spectral norms, and summarizes their temporal and cross-layer variation for lightweight supervised classification. Experiments on HaluEval, RAGTruth, and MedHallu show that DLIPS outperforms the compared baselines, including an approximately 11% relative improvement in AUROC over the strongest baseline on HaluEval. Ablation studies show that removing temporal features causes the largest F1 decrease among the evaluated feature groups, indicating that temporal Lipschitz dynamics provide information beyond overall sensitivity magnitude. Additional evaluations on different models extend the framework across model scales and architectures. These results support temporal Lipschitz dynamics as a complementary signal to representation- and consistency-based approaches for hallucination detection. DLIPS:https://anonymous.4open.science/r/DLIPS-2EB4

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, hallucination detection, Lipschitz analysis, internal state dynamics.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aJsYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DLIPS, a white-box method that detects whether a given answer is a hallucination from how sensitive the model is while reading the answer. Specifically, at each prefix of the answer, DLIPS measures how much the selected layers amplify a small change to the last token by computing the spectral norm of the layer's Jacobian. It then summarizes how these values change across prefixes to build the features for a simple classifier.

The paper claims three main contributions.

  1. It reveals that the sensitivity changes along the answer carry information for hallucination detection beyond per-layer and global statistics of the sensitivity.
  2. A detection framework that computes this sensitivity without forming the Jacobian, and classifies an answer with a frozen model.
  3. An evaluation across general QA, RAG, and medical QA tasks.

For validation, DLIPS with logistic regression reports better F1 and AUROC than the compared baselines on HaluEval, RAGTruth, and MedHallu with Qwen3.5-0.8B.

Critical strengths

The paper computes the Jacobian norm of each complete decoder block without forming the Jacobian, and checks the estimates against exact SVD (Appendix D). This supports the accuracy of the measured sensitivity values.

Critical weaknesses
  1. The performance advantage of the proposed method may still come from indirectly using the length of the LLM’s answer. Prior work shows that a simple detector using the length of response as feature alone can match complex hallucination detectors [1]. Although the paper provides an ablation study showing that the performance drop of removing the Global group that includes the length-related features is small, the Temporal features still contain length-related information:
  • For answers of 1–3 tokens, some features are always zero, e.g., all ten Temporal features are zero for a one-token answer.
  • For answers of 4–20 tokens, some features, such as the dynamic range, tend to increase along with the length.
  • For answers longer than 20 tokens, the 20 prefixes are spread across the answer, so a longer answer has larger gaps between prefixes. According to Appendix B, the sensitivity differences between neighboring prefixes are not divided by the gap, so features such as the interval-averaged total variation may also change with the length.
  1. The main results lack uncertainty estimates. Tables 1 and 2 report point estimates without variability across seeds or splits or confidence intervals, which makes it unclear whether the improvements are stable.

[1] Janiak, Denis, Jakub Binkowski, Albert Sawczyn, Bogdan Gabrys, Ravid Shwartz-Ziv, and Tomasz Jan Kajdanowicz. "The illusion of progress: Re-evaluating hallucination detection in llms." In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

Questions for authors
  1. How well does a classifier detect hallucinations if it uses only the prompt and answer lengths as input? Please compare it with DLIPS on the same test set. The released code in evaluation.py already includes a similar comparison. Please report its results.
  2. For each dataset, which subsets did you use, and how many examples were in the training, validation, and test sets?
9 comments
AuthorsOpenReview ↗

Response to Reviewer aJsY (1/3)

We sincerely thank you for your thorough review and constructive feedback. Our detailed responses are provided below.

W1: The performance advantage of the proposed method may still come from indirectly using the length of the LLM's answer. Prior work shows that a simple detector using the length of response as feature alone can match complex hallucination detectors [1]. Although the paper provides an ablation study showing that the performance drop of removing the Global group that includes the length-related features is small, the Temporal features still contain length-related information:

  • For answers of 1–3 tokens, some features are always zero, e.g., all ten Temporal features are zero for a one-token answer.
  • For answers of 4–20 tokens, some features, such as the dynamic range, tend to increase along with the length.
  • For answers longer than 20 tokens, the 20 prefixes are spread across the answer, so a longer answer has larger gaps between prefixes. According to Appendix B, the sensitivity differences between neighboring prefixes are not divided by the gap, so features such as the interval-averaged total variation may also change with the length.

A1: We agree that temporal features can, in principle, encode length through the three mechanisms described, but our results show that DLIPS's performance does not depend on them. (i) Answers of 1–3 tokens are excluded from our evaluation for all methods. (ii) On a cohort where length, checkpoint count, and checkpoint spacing are identical within each faithful/hallucinated pair, DLIPS still achieves AUROCs of 0.884/0.770/0.776 on HaluEval/RAGTruth/MedHallu. (iii) Detection errors show no monotonic trend with answer length. We detail each point below.

Answers of 1–3 tokens. DLIPS models how sensitivity evolves during generation, and temporal statistics such as first- and second-difference variability need a minimum trajectory length to be well defined. As Table R14 shows, such answers occur almost only in HaluEval (QA), mostly as single-word or yes/no answers. To align HaluEval with the length regime of the other two benchmarks, we filter out its answers shorter than five tokens before sampling (Appendix E.1). This filter was fixed before any experiments; all baselines use exactly the same filtered samples and seeded splits. We target multi-token generation, where hallucinations develop during decoding, rather than single-token answers.

Answers of 4–20 and >20 tokens. Here the effective checkpoint count (T ≤ N) and the checkpoint spacing (T > N) vary with T. To isolate these effects, we use the length-matched cohort (Appendix H.3): each hallucinated answer is paired with a faithful answer from the same QA record and prompt, with identical token and word counts and a difference of at most three characters. Because T is identical within each pair, the effective checkpoint count, checkpoint positions (also verified empirically), and inter-checkpoint gaps are identical by construction, and length is identically distributed across the two classes. These factors therefore cannot separate the classes, and any remaining separability must come from sensitivity values. Two analyses support this:

  • Length-controlled results. As shown in Table R18 (our response to Question 1), length-only detection reaches AUROCs of 0.672/0.631/0.648 on the full set, well below DLIPS (0.929/0.789/0.808). On the matched cohort (100/120/80 pairs), DLIPS retains 0.884/0.770/0.776, a drop of only 0.045/0.019/0.032, showing that it relies on sensitivity dynamics rather than length.

  • Error rates across length intervals. Table R15 reports error rates by length (Qwen3.5-0.8B, 24-feature logistic regression, up to five checkpoints, prompt-grouped folds, threshold 0.5, five seeds; Appendix H.7). Errors are not monotonic in length; they peak at 21–50 tokens for HaluEval and MedHallu and at 51–100 for RAGTruth, while the longest RAGTruth interval (>100), with the largest checkpoint gaps, has one of the lowest error rates (20.47%), contrary to a spacing-driven pattern.

Table R14: Answer-length distributions before filtering (tokens).

DatasetResponsesMedian [P25–P75]1–34–2021–5051–100>100
HaluEval (QA)20,0007 [3–14]5,22312,4072,333361
RAGTruth17,790158 [115–208]12653492,39114,784
MedHallu20,00038 [28–57]11,13812,5025,627732

Table R15: Detection error rates (%) across length intervals, averaged over five seeds (↓).

Dataset5–2021–5051–100>100
HaluEval14.6919.0314.48–
RAGTruth20.6724.1829.6320.47
MedHallu34.1740.5936.1337.67

Reference

[1] Janiak, D., Binkowski, J., Sawczyn, A., Gabrys, B., Shwartz-Ziv, R., and Kajdanowicz, T. J. "The illusion of progress: Re-evaluating hallucination detection in LLMs." EMNLP 2025.

AuthorsOpenReview ↗

Response to Reviewer aJsY (2/3)

W2: The main results lack uncertainty estimates. Tables 1 and 2 report point estimates without variability across seeds or splits or confidence intervals, which makes it unclear whether the improvements are stable.

A2: We thank the reviewer for this comment. To assess statistical reliability, we repeated the experiments in Tables 1 and 2 using five random seeds (244–248). For each seed, we randomly sampled class-balanced responses per dataset and applied identical seeded train/test splits across methods. All results are reported as mean ± sample standard deviation (SD). Due to space constraints, Table R16 presents only DLIPS variants; complete baseline comparisons will be provided in the updated manuscript Table 1.

The repeated experiments confirm the consistency of our main findings. On Qwen 3.5-0.8B, DLIPS-Logistic achieves mean F 1 scores of 0.861/0.725/0.748 and AUROCs of 0.929/0.789/0.808 on HaluEval/RAGTruth/MedHallu, respectively, maintaining the highest mean F1 among the compared methods. The 8B/9B results also exhibit relatively small variation across seeds.

The ablation findings in Table R17 remain consistent across repeated runs. Removing temporal features reduces mean F 1 by 0.127/0.032/0.042 across HaluEval/RAGTruth/MedHallu, confirming their contribution to detection performance. Separating global statistics from token-length features further clarifies their respective contributions. The middle-layer configuration consistently achieves the strongest or comparable performance, while the relatively small SDs support the stability of these observations.

We will update Table 1 with the complete five-seed results for all methods. For Table 2, we will retain the compact mean-performance comparisons and summarize the repeated-run protocol and variability in the accompanying text to accommodate space constraints.

Table R16: DLIPS detection performance across five random seeds (Acc / F 1 / AUROC, mean ± SD, ↑).

BackboneMethodHaluEvalRAGTruthMedHallu
Qwen 3.5-0.8BDLIPS-Threshold.745±.012 / .761±.010 / .823±.011.663±.010 / .660±.011 / .579±.012.546±.021 / .689±.020 / .541±.022
DLIPS-RF.857±.011 / .840±.012 / .936±.010.712±.014 / .712±.011 / .769±.012.677±.017 / .679±.016 / .734±.015
DLIPS-SVM.815±.013 / .829±.014 / .935±.012.719±.008 / .722±.007 / .785±.006.678±.025 / .702±.022 / .736±.024
DLIPS-Logistic.860±.006 / .861±.009 / .929±.007.731±.008 / .725±.007 / .789±.009.749±.011 / .748±.012 / .808±.013
Qwen 3.5-9BDLIPS-Threshold.766±.012 / .775±.011 / .827±.013.511±.010 / .657±.011 / .428±.009.640±.018 / .737±.019 / .713±.020
DLIPS-RF.891±.011 / .891±.014 / .937±.013.788±.010 / .766±.008 / .817±.009.696±.015 / .683±.014 / .756±.013
DLIPS-SVM.890±.010 / .890±.011 / .939±.013.744±.011 / .745±.010 / .827±.009.717±.011 / .707±.010 / .739±.009
DLIPS-Logistic.911±.011 / .917±.014 / .960±.009.782±.012 / .783±.011 / .864±.013.684±.013 / .678±.011 / .735±.014
Llama-3.1-8BDLIPS-Threshold.746±.010 / .772±.012 / .770±.009.489±.006 / .632±.008 / .490±.007.525±.010 / .652±.012 / .498±.011
DLIPS-RF.908±.007 / .904±.008 / .971±.006.811±.009 / .796±.007 / .856±.008.646±.023 / .632±.021 / .707±.022
DLIPS-SVM.901±.006 / .894±.009 / .973±.007.801±.009 / .765±.008 / .841±.011.699±.013 / .717±.011 / .728±.010
DLIPS-Logistic.920±.010 / .921±.008 / .981±.011.772±.006 / .754±.008 / .839±.007.706±.020 / .715±.021 / .733±.016

Table R17: Feature and layer ablations across five random seeds (F 1 / AUROC, mean ± SD, ↑).

TypeConfigurationHaluEvalRAGTruthMedHallu
FeatureFull Features.861±.009 / .929±.007.725±.007 / .789±.009.748±.012 / .808±.013
w/o Temporal.734±.007 / .866±.008.693±.012 / .749±.011.706±.012 / .769±.011
w/o Spatial.860±.013 / .920±.010.720±.014 / .801±.012.716±.011 / .762±.017
w/o Global Statistics.828±.012 / .901±.013.711±.013 / .778±.008.744±.015 / .795±.014
w/o Token Lengths.793±.006 / .887±.009.703±.009 / .768±.010.715±.017 / .778±.012
LayerFull Layers.860±.011 / .928±.009.724±.009 / .773±.010.731±.018 / .807±.020
Bottom Layers.508±.009 / .512±.010.492±.011 / .529±.012.534±.021 / .585±.024
Middle Layers.861±.009 / .929±.007.725±.007 / .789±.009.748±.012 / .808±.013
Top Layers.805±.006 / .887±.007.678±.010 / .731±.011.686±.011 / .771±.012
AuthorsOpenReview ↗

Response to Reviewer aJsY (3/3)

Q1: How well does a classifier detect hallucinations if it uses only the prompt and answer lengths as input? Please compare it with DLIPS on the same test set. The released code in evaluation. py already includes a similar comparison. Please report its results.

A3: We thank the reviewer for this suggestion. To isolate the contribution of response-length information, we evaluate w/o Global Statistics and w/o Token Lengths in Table R17. Removing explicit length features reduces AUROC by 0.042/0.021/0.030 on HaluEval/RAGTruth/MedHallu, respectively, while removing global statistics yields reductions of 0.028/0.011/0.013. These results indicate that length information contributes to detection but does not fully account for DLIPS's performance.

To further examine potential length-related shortcuts, we conducted length-only and length-matched evaluations, comparing a length-based Logistic classifier with DLIPS. Table R18 compares full DLIPS, length-only classification, and length-matched DLIPS.

Table R18: Length-only and length-matched controls (F 1 / AUROC, ↑).

DatasetFull DLIPSLength-onlyLength-matched DLIPS
HaluEval0.861 / 0.9290.601 / 0.6720.790 / 0.884
RAGTruth0.725 / 0.7890.566 / 0.6310.706 / 0.770
MedHallu0.748 / 0.8080.585 / 0.6480.713 / 0.776

The length-only classifier achieves AUROCs of 0.672/0.631/0.648, substantially below full DLIPS (0.929/0.789/0.808), corresponding to improvements of 0.257/0.158/0.160. Furthermore, on 100/120/80 length-matched pairs, the length-only AUROC drops to near chance (0.512/0.507/0.504), whereas DLIPS retains AUROCs of 0.884/0.770/0.776, correctly distinguishing hallucinated from faithful answers in 85.0%/80.8%/81.2% of pairs.

These results demonstrate that Jacobian sensitivity dynamics provide substantial discriminative information beyond response-length differences. We will incorporate the revised ablations and length-controlled comparisons into the manuscript.

Q2: For each dataset, which subsets did you use, and how many examples were in the training, validation, and test sets?

A4: We evaluated DLIPS on the QA subset of HaluEval, RAGTruth, and MedHallu, covering general question answering, retrieval-augmented generation, and medical hallucination detection, respectively. For each of the five random seeds (244–248), we independently sampled 800 class-balanced responses (400 hallucinated and 400 faithful) from each dataset. In each run, the newly sampled responses were divided into training, validation, and test sets at a ratio of 60%/20%/20% (480/160/160 examples). Splits were grouped by question or source to ensure that related responses and faithful–hallucinated pairs remained within the same partition, preventing data leakage. The same sampled responses and seeded splits were used across all compared methods within each run, and results were reported as mean ± standard deviation over the five runs. For HaluEval (QA), answers shorter than five tokens were excluded during the initial preprocessing. We will clarify the dataset selection, repeated-sampling protocol, and splitting details in the revised manuscript.


We thank you again for the thoughtful suggestions. We believe that these additional experiments and clarifications have strengthened our work and addressed the concerns raised. We would greatly appreciate any further feedback and welcome suggestions on any remaining issues.

Reviewer_XZP6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DLIPS, a supervised white-box method for response-level hallucination detection based on temporal patterns in local decoder-block sensitivity. Motivated by the possibility that final-state or globally aggregated signals may obscure transient information, the method estimates Jacobian spectral norms at selected answer prefixes using matrix-free power iteration. It summarizes these measurements into temporal, layer-wise, and global features, appends token-length metadata, and trains a lightweight classifier with a frozen backbone.

Critical strengths
  1. The overall pipeline is clearly structured. The appendices provide useful detail on motivation, feature construction and numerical estimation.
  2. The empirical evaluation covers general QA, retrieval-augmented generation, and medical QA on Qwen3.5-0.8B. DLIPS achieves the highest reported F1 among the compared methods on all three datasets, providing encouraging evidence for the usefulness of the proposed features across these settings.
Critical weaknesses
  1. DLIPS repeatedly estimates Jacobian norms across eight decoder blocks, whereas TSV uses a localized steering intervention with similar performance. The does not establish whether monitoring only a few layers can preserve DLIPS’s detection performance. This leaves the necessity of its broader internal-state analysis unclear. Although Appendix D.3 reports DLIPS’s runtime, the paper does not quantify its computational cost or peak GPU memory usage relative to the baselines, making the cost of its performance gains difficult to assess.
  2. The relation between Jacobian sensitivity and hallucination seems unnatural. The evaluation does not sufficiently isolate the contribution of Jacobian-based sensitivity from the broader temporal aggregation and supervised classification pipeline. It remains unclear whether the proposed sensitivity measurement provides additional predictive value over simpler internal signals processed through the same framework.
Questions for authors

Kindly refer to the questions mentioned in the weaknesses section above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XZP6 (1/2)

We sincerely thank you for the valuable and constructive feedback, which has helped us identify important areas for improvement. We have carefully addressed the concerns through additional experiments, analyses, and manuscript revisions. Our detailed point-by-point responses are provided below.

W1: DLIPS repeatedly estimates Jacobian norms across eight decoder blocks, whereas TSV uses a localized steering intervention with similar performance. The does not establish whether monitoring only a few layers can preserve DLIPS’s detection performance. This leaves the necessity of its broader internal-state analysis unclear. Although Appendix D.3 reports DLIPS’s runtime, the paper does not quantify its computational cost or peak GPU memory usage relative to the baselines, making the cost of its performance gains difficult to assess.

A1: We thank the reviewer for raising these important concerns regarding layer selection and computational efficiency. We conducted additional experiments to examine whether fewer decoder blocks can preserve DLIPS's detection performance and further quantified its computational overhead across different model scales.

1. Layer selection. We evaluated fixed subsets of decoder blocks using the same Qwen 3.5-0.8B sensitivity trajectories, statistical summaries, Logistic classifier, and grouped folds.

The results in Table R11 reveal a dataset-dependent trade-off. On RAGTruth, a single block retains most of the eight-block AUROC (0.676 vs. 0.746), indicating that localized sensitivity measurements can already provide useful detection signals. However, the performance gaps are larger on HaluEval (0.748 vs. 0.895) and MedHallu (0.546 vs. 0.724), suggesting that broader layer coverage can capture additional discriminative information. Eight blocks achieve the highest average AUROC among the tested configurations. Nevertheless, since both the number and selection of blocks vary, these results do not establish a universally optimal or minimum layer count.

Table R11: AUROC comparison across different decoder-block subsets (↑).

Blocks (zero-indexed)HaluEvalRAGTruthMedHallu
110.7480.6760.546
11, 150.8050.6680.584
9, 11, 13, 150.8590.6850.576
8–150.8950.7460.724

2. Computational overhead. To quantify the computational cost of DLIPS, we profiled per-block detection time and GPU memory usage across four model scales on an NVIDIA RTX PRO 6000. Table R12 reports these measurements.

The results show that the additional DLIPS memory remains modest (14–262 MiB), with total GPU memory dominated by backbone weights and caches rather than explicit Jacobian storage. However, repeated JVP/VJP computations introduce substantial runtime overhead, making DLIPS computationally more demanding than lightweight approaches such as TSV. The reported times reflect per-block detection costs rather than end-to-end latency, and therefore should not be interpreted as a direct runtime comparison with TSV.

Overall, we acknowledge that broader sensitivity monitoring is not uniformly necessary across datasets. While fewer blocks can retain competitive performance in some settings, monitoring additional layers provides measurable improvements on HaluEval and MedHallu at increased computational cost. These findings clarify the accuracy–efficiency trade-off of DLIPS rather than establishing eight-block monitoring as an indispensable configuration. We will incorporate the layer-subset analysis and computational profiling results into the revised manuscript and Appendix D.3.

Table R12: Per-block detection time and GPU memory usage across model scales.

BackboneTime per block (s)Weights + cache (GiB)Extra DLIPS memory (MiB)Total (GiB)
Qwen 3.5-0.8B0.831.42141.44
Llama-3.1-8B0.7515.0011815.11
Qwen 3.5-9B0.9616.7410416.84
Qwen 3-32B1.1161.1126261.36
AuthorsOpenReview ↗

Response to Reviewer XZP6 (2/2)

W2: The relation between Jacobian sensitivity and hallucination seems unnatural. The evaluation does not sufficiently isolate the contribution of Jacobian-based sensitivity from the broader temporal aggregation and supervised classification pipeline. It remains unclear whether the proposed sensitivity measurement provides additional predictive value over simpler internal signals processed through the same framework. We thank the reviewer for this important comment. We agree that the predictive value of Jacobian-based sensitivity should be distinguished from the benefits of temporal aggregation and supervised classification.

A2: DLIPS does not assume that high Jacobian sensitivity directly indicates hallucination. Rather, the decoder-block Jacobian spectral norm measures the maximum local amplification of input perturbations, characterizing how sensitive the model's computation is to small changes in its internal representations. Unlike hidden states or logits, which describe realized activations or output distributions, Jacobian sensitivity captures the local response of the computation to perturbations. We investigate whether its temporal variation provides discriminative information associated with hallucinated responses, without claiming a direct causal relationship.

To isolate the contribution of Jacobian sensitivity, we conducted additional experiments replacing the input signal while keeping the Qwen3.5-0.8B backbone, sampled answer prefixes, 21 statistical features, Logistic classifier, and grouped folds identical. Each dataset contains 200 evaluated responses, and explicit token-length features are excluded. Attention-based comparisons use the same two full-attention blocks (11 and 15).

As shown in Table R13, under the matched eight-block configuration, Jacobian sensitivity consistently outperforms the simpler internal signals, improving AUROC over the strongest alternative by 0.064, 0.112, and 0.117 on HaluEval, RAGTruth, and MedHallu, respectively. Since the temporal aggregation, classifier, and evaluation protocol are held fixed, these improvements support the additional predictive value of the Jacobian-based measurement rather than merely reflecting the benefits of the downstream detection pipeline.

The two-block comparison further demonstrates that Jacobian sensitivity maintains higher AUROC than attention-based statistics across all three datasets, with clear improvements on HaluEval and MedHallu and comparable performance on RAGTruth. These results further support the discriminative value of Jacobian-based sensitivity, even with reduced layer coverage.

We will incorporate these controlled comparisons into the revised manuscript and clarify that DLIPS characterizes hallucination-associated computational sensitivity patterns rather than directly measuring factual correctness or causal error propagation.

Table R13: AUROC comparison of different temporal signals under matched experimental settings (↑).

Temporal signalBlocksHaluEvalRAGTruthMedHallu
Jacobian spectral norm8–150.8850.7910.754
RMS hidden-state update8–150.7820.6790.600
RMS block output8–150.8210.5080.637
Logit-lens entropy8–150.7440.5540.584
Jacobian spectral norm11, 150.8360.7280.644
Attention log-diagonal11, 150.7690.7170.550

We thank you again for the thoughtful suggestions. We believe that these additional experiments and clarifications have strengthened our work and addressed the concerns raised. We would greatly appreciate any further feedback and welcome suggestions on any remaining issues.

I remain concerned about the computational overhead of DLIPS and whether its reported performance gains stem from the proposed structural design or from additional computation or access to richer information. The reported per-block detection times alone do not clarify the additional cost relative to the baselines. Since hallucination detection should ideally introduce only modest inference overhead, a direct runtime comparison between DLIPS and the Base/TSV baselines under identical hardware and evaluation settings would help assess its practical efficiency.

Reviewer_YFkdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DLIPS, a hallucination detection framework that leverages temporal variations in local Lipschitz sensitivity to identify hallucinated responses. Specifically, the authors first sample multiple checkpoints along a candidate answer and extract hidden states from selected decoder blocks of a frozen language model. At each checkpoint, they measure the local sensitivity of the current token's decoder-block output to perturbations of its input hidden state by estimating the Jacobian spectral norm using matrix-free power iteration with Jacobian-vector and vector-Jacobian products. The resulting sensitivity measurements are organized into layer-wise temporal trajectories, from which temporal, layer-wise, and cross-layer statistical features are extracted and combined with token-length information. These features are then fed into supervised classifiers, including logistic regression, support vector machines, and random forests, to predict whether the candidate answer contains hallucinated content. Experiments on HaluEval, RAGTruth, and MedHallu demonstrate competitive detection performance, while feature and layer ablations investigate the contributions of temporal sensitivity dynamics and the choice of monitored decoder layers. However, I have some concerns about this paper.

Critical strengths
  1. The authors propose DLIPS, a hallucination detection framework that incorporates local decoder-block Jacobian spectral norms into response-level hallucination detection, providing a sensitivity-based alternative to conventional output- and representation-based approaches.
  2. The proposed method characterizes the temporal evolution of local computational sensitivity across answer prefixes and decoder layers, enabling the extraction of temporal and cross-layer features rather than relying solely on static sensitivity measurements.
  3. The authors conduct experiments on three hallucination detection benchmarks (HaluEval, RAGTruth, and MedHallu) and multiple LLM backbones, supplemented by feature and layer ablations, numerical validation, and hyperparameter analyses to evaluate the proposed framework.
Critical weaknesses
  1. Limited methodological novelty. The proposed method mainly combines existing Jacobian spectral-norm estimation techniques with conventional temporal statistical features. Since prior studies[A][B][C] have already explored perturbation and gradient-based hallucination detection, the novelty of this work appears limited, and its incremental contributions over existing methods are not sufficiently demonstrated.
  2. Insufficient validation of the core motivation. The paper motivates DLIPS through Temporal Signal Dilution. However, Figure 1 is merely conceptual, and the paper lacks empirical comparisons across early, middle, and late hallucinations to verify this phenomenon or demonstrate that DLIPS effectively addresses it.
  3. Insufficient justification for using Jacobian spectral norms. Table 2 does not establish the advantages of Jacobian spectral norms over simpler temporal signals, such as hidden states, logits, or attention patterns. Controlled comparisons under consistent experimental settings are needed to justify this design choice.
  4. Misalignment between local Jacobian measurements and temporal error propagation. Local decoder-block Jacobians at individual answer checkpoints do not directly capture how early hallucination signals propagate or diminish during subsequent generation, leaving a gap between the method and its motivation.
  5. Limited exploration of temporal feature modeling. DLIPS relies on handcrafted statistical features that may discard important temporal information, particularly since these features are invariant to time reversal. The lack of comparisons with learnable temporal encoders, such as CNNs, GRUs, or Transformers, raises concerns about whether the proposed feature design adequately captures temporal dynamics.
  6. Incomplete baseline comparisons across model scales. Table 1 compares DLIPS with existing baselines only on Qwen3.5-0.8B, while experiments on Qwen3.5-9B and Llama-3.1-8B include only DLIPS variants. Consequently, it remains unclear whether the reported advantages over existing methods generalize to larger models and different architectures.
  7. Insufficient evaluation of scalability to larger LLMs. The experiments are limited to models of up to 9B parameters, without evaluating larger models such as 32B. Given the repeated JVP/VJP computations required by DLIPS, the lack of systematic runtime, memory, and performance–cost analyses across model scales makes its practical scalability unclear.
  8. Insufficient statistical reliability analysis. The paper reports performance on a single train/validation/test split without statistical significance tests. Repeated group-wise data splits or paired bootstrap analyses would help establish whether the reported improvements over baselines are statistically robust.
  9. Insufficient control for potential dataset biases and feature redundancy. The ablation study in Table 2 does not isolate the contribution of token-length features, as the w/o Global setting removes both global statistics and length features. Furthermore, the lack of length-only and length-matched controls leaves open whether dataset biases or response-length differences contribute to the reported improvements.
  10. The Related Work section overlooks several relevant studies on hallucination detection, including [A–D]. The authors are encouraged to discuss these works and clarify how DLIPS differs from existing perturbation-based, gradient-based, and zero-source hallucination detection approaches.

Reference

[A] Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations. 2025 arXiv.

[B] From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity. 2026 ICML.

[C] Grad Detect: Gradient-Based Hallucination Detection in LLMs. 2026 arXiv.

[D] Zero-source LLM Hallucination Detection with Human-like Criteria Probing. 2026 ICML.

Questions for authors

NA

10 comments
AuthorsOpenReview ↗

Response to Reviewer YFkd (1/8)

We sincerely thank you for the valuable and constructive feedback, which has helped us identify important areas for improvement. We have carefully addressed the concerns through additional experiments, analyses, and manuscript revisions. Our detailed point-by-point responses are provided below.

W1: Limited methodological novelty. The proposed method mainly combines existing Jacobian spectral-norm estimation techniques with conventional temporal statistical features. Since prior studies[A][B][C] have already explored perturbation and gradient-based hallucination detection, the novelty of this work appears limited, and its incremental contributions over existing methods are not sufficiently demonstrated.

A1: We thank the reviewer for this important comment. While matrix-free power iteration is established, DLIPS contributes a distinct sensitivity-based temporal characterization with two empirically demonstrated benefits: improved detection performance over existing methods and informative hallucination signals from early answer prefixes.

Unlike SSP [A], which measures perturbation-induced representation shifts, EPGS [B], which examines embedding-perturbed gradient magnitudes, Grad Detect [C], which uses layer-wise gradients, and HCPD [D], which employs LLM-generated evaluation criteria, DLIPS tracks the spectral norm of the complete decoder-block input–output Jacobian across answer prefixes, including residual pathways. This captures maximal local perturbation amplification without modifying the input or relying on a detection loss.

To quantify these contributions, we evaluated all methods on the same response subset per dataset at 25%, 50%, 75%, and 100% of answer tokens (Table R1). DLIPS achieves the highest AUROC in 10 of 12 settings, with one tie and one exception (MedHallu at 100%). Notably, at only 25% of the answer, DLIPS already achieves AUROCs of 0.913/0.758/0.708 on HaluEval/RAGTruth/MedHallu, outperforming the strongest competing methods by 0.128/0.086/0.065, respectively. These findings demonstrate both the incremental predictive value of our approach and its ability to capture hallucination-associated sensitivity patterns well before answer completion.

Table R1: Performance comparison of extracting response results at different answer positions and performing detection

DatasetMethod25%50%75%100%
HaluEvalDLIPS0.9130.8900.8980.925
SSP [A]0.4930.5180.5130.584
EPGS [B]0.6420.7570.8310.857
Grad Detect [C]0.6740.8250.8520.872
HCPD [D], 9 B judge0.7850.8380.8350.848
PPL0.6360.5640.5800.609
LLMCheck (HS)0.5420.5740.6370.621
LLMCheck (Attn)0.7330.8050.7790.767
TOHA0.4660.6410.7040.757
RAGTruthDLIPS0.7580.7510.7640.780
SSP [A]0.5330.6190.6280.663
EPGS [B]0.5210.4560.4830.529
Grad Detect [C]0.5840.4450.4230.635
HCPD [D], 9 B judge0.5840.6630.7390.748
PPL0.4960.5940.6130.647
LLMCheck (HS)0.6720.6920.6680.697
LLMCheck (Attn)0.6680.7350.7360.780
TOHA0.4620.6010.6710.718
MedHalluDLIPS0.7080.7120.7260.775
SSP [A]0.5690.5540.5840.543
EPGS [B]0.5920.5140.5690.617
Grad Detect [C]0.6160.4960.5860.624
HCPD [D], 9 B judge0.5840.6790.7190.786
PPL0.3940.3460.3260.329
LLMCheck (HS)0.4210.4590.4220.544
LLMCheck (Attn)0.6390.6650.6150.706
TOHA0.6430.6920.6770.698

Reference

[A] Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations. 2025 arXiv.

[B] From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity. 2026 ICML.

[C] Grad Detect: Gradient-Based Hallucination Detection in LLMs. 2026 arXiv.

[D] Zero-source LLM Hallucination Detection with Human-like Criteria Probing. 2026 ICML.

AuthorsOpenReview ↗

Response to Reviewer YFkd (2/8)

W2: Insufficient validation of the core motivation. The paper motivates DLIPS through Temporal Signal Dilution. However, Figure 1 is merely conceptual, and the paper lacks empirical comparisons across early, middle, and late hallucinations to verify this phenomenon or demonstrate that DLIPS effectively addresses it.

A2: We thank the reviewer for highlighting the need for empirical validation beyond the conceptual illustration in Figure 1. To investigate temporal signal dilution, we manually curated 100 hallucinated–faithful response pairs from RAGTruth, categorized by the first annotated hallucination position (33 early, 36 middle, and 31 late), with faithful controls matched by task and approximate answer length. We compared four detectors: PPL, LLMCheck-HS, a final-checkpoint Jacobian detector, and DLIPS using multiple Jacobian checkpoints. Both Jacobian variants used the same eight blocks, Logistic regression, grouped splits, and no explicit length features.

The results in Table R2 support our motivation in two ways. First, all three completed-answer baselines show higher FNR for early-onset hallucinations than for middle- or late-onset errors, providing empirical evidence consistent with temporal signal dilution. Second, by retaining local Jacobian sensitivity information across answer prefixes, DLIPS reduces early-onset FNR from 63.6% to 36.4% (−27.2 percentage points), recovering 9 of the 21 early hallucinations missed by the final-checkpoint detector. Its FNR also remains consistent across early, middle, and late errors (36.4%/36.1%/35.5%). These results demonstrate how multi-prefix sensitivity analysis mitigates the loss of early hallucination evidence in final-checkpoint detection. We will incorporate this analysis alongside Figure 1.

Table R2: False negative rates (FNR, ↓) across hallucination onset positions on 100 matched RAGTruth pairs.

First error onsetPairsPPLLLMCheck-HSFinal JacobianDLIPS (Multi-prefix)
Early3351.5%45.5%63.6%36.4%
Middle3644.4%36.1%41.7%36.1%
Late3141.9%38.7%41.9%35.5%


W3: Insufficient justification for using Jacobian spectral norms. Table 2 does not establish the advantages of Jacobian spectral norms over simpler temporal signals, such as hidden states, logits, or attention patterns. Controlled comparisons under consistent experimental settings are needed to justify this design choice.

A3: To isolate the contribution of Jacobian spectral norms, we conducted controlled comparisons by replacing the input signal while keeping the Qwen 3.5-0.8B backbone, sampled prefixes, 21 statistical features, Logistic classifier, and grouped folds identical (200 responses per dataset, without explicit length features). Attention-based comparisons use the same two full-attention blocks (11 and 15).

As shown in Table R3, the eight-block Jacobian signal consistently outperforms all matched eight-block alternatives, improving AUROC over the strongest alternative by 0.064, 0.112, and 0.117 on HaluEval, RAGTruth, and MedHallu, respectively. Unlike hidden-state updates or output magnitudes, which characterize realized activations, the Jacobian spectral norm measures the maximum local amplification of input perturbations, capturing a complementary property of the model's computation. Logit entropy reflects output uncertainty, while attention statistics characterize attention distributions rather than complete block sensitivity. The two-block comparison further demonstrates that Jacobian sensitivity maintains higher AUROC than attention-based statistics across all three datasets, with clear improvements on HaluEval and MedHallu and comparable performance on RAGTruth. These results further support the discriminative value of Jacobian-based sensitivity, even with reduced layer coverage. Overall, these controlled results support local Jacobian sensitivity as a more informative temporal signal than the evaluated simpler alternatives, rather than merely benefiting from temporal aggregation or classifier choice.

Table R3: AUROC comparison of different temporal signals under matched experimental settings (↑).

Temporal signalBlocksHaluEvalRAGTruthMedHallu
Jacobian spectral norm8–150.8850.7910.754
RMS hidden-state update8–150.7820.6790.600
RMS block output8–150.8210.5080.637
Logit-lens entropy8–150.7440.5540.584
Jacobian spectral norm11, 150.8360.7280.644
Attention log-diagonal11, 150.7690.7170.550
AuthorsOpenReview ↗

Response to Reviewer YFkd (3/8)

W4: Misalignment between local Jacobian measurements and temporal error propagation. Local decoder-block Jacobians at individual answer checkpoints do not directly capture how early hallucination signals propagate or diminish during subsequent generation, leaving a gap between the method and its motivation.

A4: We apologize for not making this distinction sufficiently clear in the manuscript. DLIPS does not directly measure the causal propagation of an early hallucination through subsequent generation. Instead, Equation (4) quantifies the local input–output sensitivity of each decoder block at a given answer prefix, conditioned on the preceding context. By repeating this measurement across multiple answer-prefix checkpoints and decoder layers, DLIPS constructs a layer-by-checkpoint sensitivity matrix, from which temporal and cross-layer features characterize how local computational sensitivity evolves throughout the answer.

Thus, while individual Jacobian measurements capture local sensitivity rather than causal error propagation, their temporal aggregation enables DLIPS to retain hallucination-associated signals that may be less apparent at the final checkpoint. This is supported by our onset-stratified experiments in Response W2 (Table R2), where multi-prefix DLIPS reduces the early-onset false negative rate from 63.6% to 36.4% compared with final-checkpoint Jacobian detection. We will clarify this distinction and the scope of our claims alongside Equation (4).



W5: Limited exploration of temporal feature modeling. DLIPS relies on handcrafted statistical features that may discard important temporal information, particularly since these features are invariant to time reversal. The lack of comparisons with learnable temporal encoders, such as CNNs, GRUs, or Transformers, raises concerns about whether the proposed feature design adequately captures temporal dynamics.

A5: We acknowledge that our statistical features are invariant to time reversal. However, DLIPS focuses on response-level hallucination detection rather than pinpointing the exact onset or direction of error propagation. Our evaluation at different answer stages (Response W1, Table R1) instead examines whether the available answer prefixes already contain discriminative hallucination-related patterns. Accordingly, our features emphasize the magnitude, variability, and cross-layer patterns of sensitivity fluctuations, which can provide effective detection signals without explicitly encoding temporal direction.

To assess whether learnable temporal modeling provides additional benefits, we compared our statistical summaries with a CNN applied to the same ordered eight-block sensitivity trajectories. The CNN achieved AUROCs of 0.883/0.782/0.671, compared with 0.895/0.794/0.788 for our statistical features on HaluEval/RAGTruth/MedHallu, respectively. Our approach consistently outperforms the CNN, with a particularly pronounced 0.117 AUROC advantage on MedHallu. While the CNN must learn temporal representations from task-specific training data, DLIPS uses predefined statistical summaries with a lightweight classifier, reducing temporal modeling complexity. These results suggest that the magnitude and variability of sensitivity dynamics provide robust discriminative information without requiring a learned temporal encoder, particularly in specialized domains such as medical QA.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers, ACs, SACs, and PCs

We sincerely thank the reviewers and the ACs, SACs, and PCs for their careful evaluation and constructive suggestions. We appreciate the reviewers’ recognition of our sensitivity-based approach and temporal characterization (YFkd), clear framework and strong cross-domain performance (XZP6), and accurate, numerically validated Jacobian estimation (aJsY), as well as all three for suggestions on points the paper did not fully address. Our responses strengthen the central contribution. Temporal Lipschitz dynamics provide discriminative signals for accurate hallucination detection beyond simpler internal signals and answer-length features. Additional experiments validate this contribution and demonstrate the accuracy–latency trade-off between full-answer detection and early prediction. Revisions are highlighted in blue.

  1. Contribution of temporal Lipschitz dynamics (Reviewers YFkd and XZP6). DLIPS tracks local sensitivity across answer prefixes through complete decoder-block Jacobians. We clarified its distinction from existing perturbation- and gradient-based detectors and added SSP, EPGS, Grad Detect, and HCPD comparisons. With the backbone, prefixes, features, classifier, and grouped folds fixed, and explicit length inputs excluded, Jacobian sensitivity improves AUROC over the strongest matched alternative by 0.064/0.112/0.117 on HaluEval/RAGTruth/MedHallu. Our statistical temporal features also outperform temporal encoder models such as CNNs, achieving 0.895/0.794/0.788 AUROC versus 0.883/0.782/0.671. These results establish the predictive value of both the sensitivity measurement and its temporal features. See Section 2, Table 1, and Appendices H.1–H.2.

  2. Empirical validation of temporal signal dilution (Reviewer YFkd). We constructed 100 matched RAGTruth pairs stratified by first hallucination onset. PPL, LLMCheck-HS, and final-checkpoint Jacobian detection all exhibit their highest false negative rates on early hallucinations, consistent with temporal signal dilution. Multi-prefix DLIPS reduces early-onset FNR from 63.6% to 36.4%, recovering nine of 21 missed cases, and maintains FNRs of 36.4%/36.1%/35.5% across early, middle, and late hallucinations. We clarified that each Jacobian measures local sensitivity conditioned on the prefix. Temporal aggregation preserves hallucination evidence that can become less apparent at the final checkpoint. See Figure 1 and Appendix H.2.

  3. End-to-end efficiency and early prediction (Reviewers XZP6 and YFkd). Using identical prompts and matched output lengths, we measured time from prompt submission to answer completion or score availability. Full-answer DLIPS achieves 0.783/0.779 AUROC versus TSV’s 0.623/0.631 on shorter/longer responses, with end-to-end times of 18.03/33.04 seconds versus 9.55/25.18 seconds. At 25% of the answer, DLIPS retains 0.756/0.759 AUROC, exceeding full-answer TSV while adding 1.41 seconds for shorter responses and providing predictions 10.67 seconds earlier for longer responses. DLIPS therefore supports accurate full-answer detection and early risk prediction, enabling intervention before generation finishes. See Table R19 in our follow-up response to Reviewer XZP6.

  4. Predictive value beyond answer length (Reviewers YFkd and aJsY). We separated global-statistic and token-length ablations and added length-only and length-matched controls. On pairs with identical prompts, answer token and word counts, and checkpoint positions, DLIPS retains 0.884/0.770/0.776 AUROC, while length-only classification yields 0.512/0.507/0.504. These results establish discriminative information beyond length differences. Filtering criteria, checkpoint spacing, and length-stratified analyses appear in Section 6.2, Appendix E.1, and Appendices H.3 and H.7.

  5. Consistency across repeated evaluations (Reviewers YFkd and aJsY). Five independently sampled cohorts use shared grouped splits across methods. DLIPS-Logistic achieves the highest mean F1 among compared methods on all three datasets, at 0.861/0.725/0.748. Removing temporal features consistently reduces mean F1, confirming their contribution. Means, sample standard deviations, and evaluation details appear in Table 1 and Appendices E and H.4.

  6. Applicability across architectures and scales (Reviewers YFkd and XZP6). Expanded comparisons cover Qwen3.5-9B, Llama-3.1-8B, and Qwen3-32B. On 32B, DLIPS achieves 0.955/0.894/0.724 AUROC, exceeding the strongest listed comparator on every dataset. Block-subset evaluations quantify the accuracy–coverage trade-off, while profiling reports 14–262 MB of additional peak allocation per local block measurement. Together with the end-to-end comparison, these results identify operating choices across accuracy, prediction timing, and computational budgets. See Section 6.1 and Appendices H.3 and H.5–H.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.