- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether internal gradient information can be used to detect unreliable LLM responses and predict abstention. It proposes Grad Detect, which extracts layer-wise gradient features using fixed probe responses and compares them with class-specific reference gradients before feeding the resulting features into a Transformer encoder. Experiments across eleven instruction-tuned models from four model families and four QA benchmarks show promising results on correctness detection and abstention prediction. Overall, the paper provides empirical evidence for gradient-based reliability prediction, but its novelty over existing gradient-based detection approaches remains unclear, and the current evaluation does not sufficiently establish its ability to detect hallucinations in specific generated responses.
Critical strengths
- The paper addresses a practically relevant problem by investigating whether internal gradient information can provide complementary signals for detecting unreliable LLM responses and predicting abstention, offering a useful perspective beyond conventional confidence-based methods.
- The paper provides empirical evaluation across eleven instruction-tuned models from four model families and four QA benchmarks, covering correctness detection, abstention prediction, and three-way classification. The layer ablations and inference-time analyses further provide useful evidence on the trade-off between detection performance and computational overhead.
Critical weaknesses
- The methodological novelty of Grad Detect remains unclear. NGS-HD [A] has already explored a similar gradient-based detection paradigm, using gradient-derived signals and reference distributions of truthful and hallucinated tokens to assess response reliability. Grad Detect follows a conceptually similar approach by extracting gradient features, constructing class-specific reference gradients, and comparing test samples against these references for classification. Although the two methods differ in the choice of gradients and comparison mechanisms, the paper does not sufficiently establish how Grad Detect advances beyond this existing paradigm.
- The detector is not conditioned on the actual generated response. Grad Detect extracts features from fixed probe responses ("Sure" and "Unfortunately"), while its labels depend on the actual generated answers. It therefore appears to predict query-level correctness under greedy decoding rather than detect hallucinations in specific responses, limiting the scope of its claims.
- The source of the discriminative gradient signal is not sufficiently isolated. Since gradients are extracted from queries paired with fixed probe responses, they may primarily reflect question difficulty, knowledge familiarity, or answerability rather than hallucination-specific information. The paper lacks controlled analyses to establish that the proposed features provide predictive value beyond these query-level factors.
- Dependence on labeled reference data limits practical applicability. Grad Detect requires labeled examples to construct class-specific reference gradients and train a separate predictor. This may introduce an additional annotation burden, particularly in domains where reliable labels are difficult or expensive to obtain.
- The method has limited transferability across models and datasets. Reference gradients and predictors must be fitted separately for each target model, while cross-dataset transfer incurs an average accuracy drop of approximately 2.9–3.8 percentage points. These requirements limit the method's applicability to new models and domains without additional setup.
- The baseline comparison seems not fully isolate the benefit of gradient features. Grad Detect uses layer-wise features and a Transformer encoder, whereas the activation-based baseline uses final-layer hidden states and an MLP. The performance gap may therefore reflect differences in feature coverage and classifier capacity rather than the intrinsic value of gradient information.
- The comparison with existing gradient-based methods is incomplete. Although the paper discusses related gradient-based approaches, the experiments do not include direct comparisons with these methods. Evaluating Grad Detect against closely related approaches under comparable settings would help clarify its relative advantages and provide stronger evidence for its methodological contribution.
- The baseline comparison is limited in scope. The main comparison focuses on correctness detection on TriviaQA, leaving the relative performance of Grad Detect on other benchmarks and on abstention and three-way classification insufficiently established. Consequently, the reported results may not fully reflect its relative performance across the broader range of datasets and prediction tasks considered in the paper.
- The choices of gradient parameters and prediction architecture are insufficiently justified. The paper focuses on MLP down-projection gradients without systematically comparing them with gradients from other parameter groups, and the advantage of the Transformer encoder over simpler classifiers using the same features remains unclear.
[A] Next-Token Gradient Sensitivity Probing for LLM Hallucination Detection. 2025
Questions for authors
NA