acceptodds
Withdrawn from ICLR 2027

Gradients Know Best: Hallucination and Abstention Detection in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet they remain prone to generating hallucinations. Detecting these hallucinations is critical for deploying LLMs reliably in high-stakes applications. We present Grad Detect, a gradient-based approach for predicting hallucinations by analyzing layer-wise gradient patterns from a single forward-backward pass during inference. Our method shows that the internal gradient structure of a model carries rich information about the correctness of its output. This information is not accessible through output-level signals alone. We evaluate Grad Detect on several Q&A benchmarks across both hallucination detection and model abstention prediction, where it consistently outperforms confidence-based and sampling-based baselines. Through comprehensive layer ablation studies across all eleven models from four architectural families, we find that the signal is distributed across the network: any five layers retain about 97% of full accuracy, and restricting gradients to the final five costs only 0.4–1.3 points, enabling efficient deployment. Grad Detect provides a unified framework for predicting multiple dimensions of LLM reliability, offering strong predictive performance alongside interpretable insights into where and how model failures originate.

Primary area:
interpretability and explainable AI
Keywords:
hallucination detection, large language models, gradients, uncertainty quantification, abstention, interpretability, internal states, layer-wise analysis, question answering, trustworthy machine learning, LLM reliability, single-pass inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bzLsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether internal gradient information can be used to detect unreliable LLM responses and predict abstention. It proposes Grad Detect, which extracts layer-wise gradient features using fixed probe responses and compares them with class-specific reference gradients before feeding the resulting features into a Transformer encoder. Experiments across eleven instruction-tuned models from four model families and four QA benchmarks show promising results on correctness detection and abstention prediction. Overall, the paper provides empirical evidence for gradient-based reliability prediction, but its novelty over existing gradient-based detection approaches remains unclear, and the current evaluation does not sufficiently establish its ability to detect hallucinations in specific generated responses.

Critical strengths
  1. The paper addresses a practically relevant problem by investigating whether internal gradient information can provide complementary signals for detecting unreliable LLM responses and predicting abstention, offering a useful perspective beyond conventional confidence-based methods.
  2. The paper provides empirical evaluation across eleven instruction-tuned models from four model families and four QA benchmarks, covering correctness detection, abstention prediction, and three-way classification. The layer ablations and inference-time analyses further provide useful evidence on the trade-off between detection performance and computational overhead.
Critical weaknesses
  1. The methodological novelty of Grad Detect remains unclear. NGS-HD [A] has already explored a similar gradient-based detection paradigm, using gradient-derived signals and reference distributions of truthful and hallucinated tokens to assess response reliability. Grad Detect follows a conceptually similar approach by extracting gradient features, constructing class-specific reference gradients, and comparing test samples against these references for classification. Although the two methods differ in the choice of gradients and comparison mechanisms, the paper does not sufficiently establish how Grad Detect advances beyond this existing paradigm.
  2. The detector is not conditioned on the actual generated response. Grad Detect extracts features from fixed probe responses ("Sure" and "Unfortunately"), while its labels depend on the actual generated answers. It therefore appears to predict query-level correctness under greedy decoding rather than detect hallucinations in specific responses, limiting the scope of its claims.
  3. The source of the discriminative gradient signal is not sufficiently isolated. Since gradients are extracted from queries paired with fixed probe responses, they may primarily reflect question difficulty, knowledge familiarity, or answerability rather than hallucination-specific information. The paper lacks controlled analyses to establish that the proposed features provide predictive value beyond these query-level factors.
  4. Dependence on labeled reference data limits practical applicability. Grad Detect requires labeled examples to construct class-specific reference gradients and train a separate predictor. This may introduce an additional annotation burden, particularly in domains where reliable labels are difficult or expensive to obtain.
  5. The method has limited transferability across models and datasets. Reference gradients and predictors must be fitted separately for each target model, while cross-dataset transfer incurs an average accuracy drop of approximately 2.9–3.8 percentage points. These requirements limit the method's applicability to new models and domains without additional setup.
  6. The baseline comparison seems not fully isolate the benefit of gradient features. Grad Detect uses layer-wise features and a Transformer encoder, whereas the activation-based baseline uses final-layer hidden states and an MLP. The performance gap may therefore reflect differences in feature coverage and classifier capacity rather than the intrinsic value of gradient information.
  7. The comparison with existing gradient-based methods is incomplete. Although the paper discusses related gradient-based approaches, the experiments do not include direct comparisons with these methods. Evaluating Grad Detect against closely related approaches under comparable settings would help clarify its relative advantages and provide stronger evidence for its methodological contribution.
  8. The baseline comparison is limited in scope. The main comparison focuses on correctness detection on TriviaQA, leaving the relative performance of Grad Detect on other benchmarks and on abstention and three-way classification insufficiently established. Consequently, the reported results may not fully reflect its relative performance across the broader range of datasets and prediction tasks considered in the paper.
  9. The choices of gradient parameters and prediction architecture are insufficiently justified. The paper focuses on MLP down-projection gradients without systematically comparing them with gradients from other parameter groups, and the advantage of the Transformer encoder over simpler classifiers using the same features remains unclear.

[A] Next-Token Gradient Sensitivity Probing for LLM Hallucination Detection. 2025

Questions for authors

NA

Reviewer_woiPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Grad Detect, which predicts correctness and abstention from gradients computed using fixed probe responses, i.e., Sure, Unfortunately. Layer-wise cosine similarities to class-specific reference gradients feed a lightweight transformer predictor. Experiments cover eleven models and four question-answering datasets, with comparisons against confidence, activation probing, and sampling methods. The paper also studies layer selection and cross-dataset transfer.

Critical strengths
  • Lots of experiments, covering 11 models, and 4 dense model families and four datasets. The reported improvements compared with existing works show that Grad Detect performs better in the factual QA outcomes.
  • Transferring both references and the predictor across factual QA datasets is a useful test beyond in-domain fitting.
Critical weaknesses
  • Architectural generalization remains limited: All eleven evaluated models are dense, the largest is 12B, and timing is measured on a 3B model. Section 5 appropriately identifies MoE models as an open question. Expert routing could affect gradient sparsity, prototype averaging, and cosine similarity, so the current results do not establish MoE applicability. A representative MoE experiment would strengthen the evidence
  • QA-only evaluation: All experiments use TriviaQA, SciQ, PopQA, and TruthfulQA. Cross-dataset transfer therefore tests reuse within factual QA, rather than generalization to claim verification or factuality assessment of longer generated text. It would be nice authors can add more more dataset evaluation from fact-checking which is more directly connecting to hallucination.
  • Evaluation partitions and label construction remain insufficiently specified.
Questions for authors
  • MoE: Could the authors add an experiment on a representative open-weight MoE model, explaining how expert-specific gradients are aggregated and how routing affects prototype construction, predictive accuracy, and memory/runtime overhead?
  • Instead of QA-only dataset evaluation, can we introduce some experiments on fact checking benchmarks, and see how Grad Detect performs.

If authors address the questions above, I'd like to improve my score.

Reviewer_8rsjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Grad Detect, a method for predicting hallucinations and abstention using an LLM's internal gradients. The method computes layer-wise parameter gradients using fixed probe responses, measures their cosine similarity to precomputed class-specific reference gradients, and feeds these features into a lightweight Transformer encoder to predict response correctness or abstention. Experiments cover 11 language models and four question-answering datasets, reporting improvements over confidence-based methods, sampling-based consistency methods, and hidden-state probing. Layer ablations examine the distribution of gradient information across the network and the method's computational efficiency.

Critical strengths

S1. The approach has a reasonable rationale.

Using gradients to analyze a model's internal state is a reasonable approach. Class-specific reference gradients and layer-wise similarities provide a straightforward way to convert high-dimensional parameter gradients into low-dimensional features for predicting model behavior.

S2. The layer ablations and efficiency analysis are useful.

The authors analyze how gradients from different layers affect detection performance across models. They find that using only the last five layers preserves most of the performance while reducing the additional inference cost (Section 4.4; Appendices G and H). These results provide useful empirical support for the method's computational efficiency.

Critical weaknesses

W1. The methodological novelty is limited, and the differences from prior gradient-based detection methods are not substantial enough.

There is already considerable prior work on gradient-based detection methods for LLMs. For example, GradSafe (Xie et al., ACL 2024) computes gradients using a fixed response and uses gradient similarities to detect unsafe prompts. Embedding and Gradient Say Wrong (Hu et al., EMNLP 2024) has also applied gradient information to hallucination detection.

Relative to these works, the main changes are the construction of class-specific reference gradients, the retention of layer-wise similarity features, and the use of a Transformer encoder for classification. Although the specific designs differ, these changes look more like extensions and combinations of existing ideas. The paper does not sufficiently establish what substantive new capabilities or insights these designs provide. I therefore find the overall methodological novelty limited.

W2. Missing experimental controls prevent a clear assessment of the method's contribution.

The paper primarily compares against methods such as confidence, perplexity, self-consistency, and semantic entropy, but the comparisons with more directly relevant methods based on internal features are insufficiently controlled. For example, the Internal State Probing baseline uses only the final layer's hidden states and an MLP (Section 4.3), whereas Grad Detect uses gradient features from multiple layers and a Transformer encoder.

This comparison changes the feature type, the number of layers used, and the classifier architecture simultaneously. It is therefore difficult to determine whether the gains come from gradient information itself or from richer features across layers and the classifier design. The paper also lacks direct comparisons with existing gradient-based hallucination detectors and sufficient ablations to establish the need for the proposed classifier architecture.

Without these comparisons, the experiments do not clearly establish the method's contribution beyond existing approaches.

Overall, the paper has a reasonable motivation and provides layer-wise analyses and efficiency experiments. However, the method is closely related to existing gradient-based detection research, and the contribution of the new design choices is limited. The absence of key comparisons with related methods and matched internal-feature baselines also leaves the source of the reported performance gains unclear. Considering the limited methodological novelty and insufficient experimental validation, my rating is 1.

Questions for authors

Q1. Could the authors clarify the core differences between this method and prior gradient-based detection methods such as GradSafe (Xie et al., ACL 2024) and Embedding and Gradient Say Wrong (Hu et al., EMNLP 2024)? Beyond class-specific reference gradients and the combination of layer-wise features, does the method provide detection capabilities or new findings that existing methods do not offer?

Q2. Could the authors provide a more controlled comparison of internal features, for example by using hidden states from multiple layers with the same or a comparable classifier architecture as Grad Detect? Do gradient features retain a substantial advantage when the number of layers used, classifier capacity, and training data are controlled?

Q3. The method feeds layer-wise gradient similarities into a Transformer encoder, but the paper lacks an ablation of the classifier architecture itself. What performance can a simple linear classifier or MLP achieve using the same gradient features? How much additional improvement comes from the cross-layer interactions modeled by the Transformer encoder?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.