acceptodds
Withdrawn from ICLR 2027

Gray-Box Hallucination Detection in LLMs with Token Identities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Large language models often produce fluent but factually incorrect output. Gray-box detectors identify such hallucinations from the token-probability statistics emitted during generation, avoiding the need for white-box access to hidden states that frontier models do not provide. These detectors, however, typically rely only on probability-derived representations and ignore the identities of the candidate tokens. They can therefore observe how uncertain the model is, but not what it is uncertain about. Consequently, they cannot distinguish between a model that is torn between "big" and "small" and one that is torn between "big" and "huge" when the two pairs receive the same probability distribution. We introduce GURI (Gray-box Uncertainty Representations with Identity), a lightweight gray-box hallucination detector that combines candidate-token probabilities with per-token vector identifiers. This exposes which alternatives the model considered while preserving the accessibility and transferability of the gray-box setting. Across three open-weight models and three question-answering benchmarks, GURI increases mean test ROC-AUC from 72.09% to 79.09% over a probability-only baseline. On the closed-source GPT-4.1 model, mean ROC-AUC increases from 67.39% to 71.84%. Among the three open-weight models, GURI outperforms transferred probability-only baseline in 17 of 18 directed cross-model transfers without fine-tuning. For the three open-weight models, cross-dataset transfer between HotpotQA and TriviaQA with the model held fixed improves mean ROC-AUC over transferred LOS-Net by 3.76 points.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Hallucination Detection, Large Language Models, Misbehaviors, LLM Monitors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GBfkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on hallucination detection in LLM outputs in a gray-box regime, where models are hosted behind an external API and expose only the log-probabilities of the top-K tokens. The authors propose a supervised method that converts raw probabilities and token indices into vectors and predicts correctness from this information. They provide experiments with 3 open-weight and 1 closed-weight models on several QA datasets. Additionally, the paper presents out-of-distribution experiments with model shift and dataset shift, and ablates several components of the method.

Critical strengths
  1. The paper proposes a gray-box approach to hallucination detection in LLMs that can be applied to API-based systems. The method outperforms the considered baselines and shows good transferability between models.
Critical weaknesses
  1. The paper claims its main contribution is token-identity signals, presented as a missing signal in probability-only gray-box detection. This claim is factually incorrect. Several existing methods exploit the same idea, e.g., CCP, which is cited in the work but not connected to the proposed approach, and STC [1]. Both explicitly integrate the semantics of alternative tokens.
  2. The retokenization procedure is also concerning. How did you ensure that every token in the source model’s vocabulary exists in the phi-4 tokenizer, either as a token or as a sequence of subtokens? Additionally, only one external tokenizer is tested, which raises questions about the transferability of the method.
  3. The fixed random code table is another concern: it is a major component of the method, yet its role is unclear. The fact that pre-trained codes do not outperform fixed random codes suggests that the method gains nothing from semantics. It simply uses additional information for learning, which is unrelated to the core idea of the paper. Why are the codes fixed rather than trained? How stable are they, e.g., did you test different initializations?
  4. The experimental evaluation is weak. First, the paper omits numerous baselines: seq.prob (the sum of log-probabilities), mean token entropy [2], self-certainty [3], CCP, STC, as well as sampling-based [4,5] and black-box methods [6]. Second, the benchmarks are limited to QA tasks only. Third, the paper uses lexical matching to obtain correctness labels, which is an unreliable criterion [7]. LLM-as-a-judge should be used instead.
  5. Transfer works only under model shift, while results under dataset shift are considerably weaker, even though the task (QA) remains the same. Transferability is therefore still limited.
  6. Why is ATP not the highest-scoring component in TDS? Was sampling used during generation? If so, this needs to be clarified. Moreover, according to the ablation study, the ATP component is useless and even degrades performance, so the authors should clarify why it is used in the method. Finally, even when sampling is used, ATP should in most cases be included in TDS, so this performance degradation is what one would expect from the definition.
  7. Minor issues:
    1. The notation is confusing. If ct,jc_{t,j} is the candidate token at generation step tt, what does pt(ct,j)p_t(c_{t,j}) mean? The index tt appears twice. Also, Xt,jK=pt(ct,j)X^K_{t,j} = p_t(c_{t,j}), but what is KK, and why does it disappear on the right-hand side? Overall, why are probabilities denoted by both XX and aa?
    2. Line 178: missing period.

[1] Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models (Cao et al., EACL 2026)
[2] Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph (Vashurin et al., TACL 2025)
[3] Scalable Best-of-N Selection for Large Language Models via Self-Certainty. (Z. Kang et al., NeurIPS 2025)
[4] Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space. (Qiu et al., NeurIPS 2024)
[5] Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency (Vashurin et al., NeurIPS 2025)
[6] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (Tian et al., EMNLP 2023)
[7] Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results (Santilli et al., ACL 2025)

Questions for authors

See weaknesses 2,3, and 6.

Reviewer_TtbrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Large language models often produce fluent but factually incorrect output, known as hallucination. Black-box, gray-box, and white-box methods are used for detecting hallucinations. This paper presents a gray-box detector, that uses token identify along with token probability to detect hallucinations. Their GURI (Gray-box Uncertainty Representations with Identity) method combines candidate-token probabilities with per-token vector identifiers using a lightweight detector architecture, which was trained with part of the data in the benchmarks. GURI method performed significantly better than the baseline method.

Critical strengths
  1. The posed method integrated token ids with token probabilities to detect hallucinations.
  2. The integration is done by training a lightweight fusion model.
  3. The idea of integration is novel and interesting.
Critical weaknesses

1 The training of the lightweight fusion model. It was trained with part of the data in the benchmark, which raises the question of data leakage. 2. The paper has not shown how sizes of the partition of training and evaluation datasets affect performance.

Questions for authors

Why was the description of training procedure of the fusion model not presented in the main body of the paper? Why was the effect of benchmark data splits not performed and reported?

Reviewer_HdxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GURI a grey box method for hallucination detection. GURI uses probability-derived representations, with the key novelty being the use of candidate token identities. The GURI method shows promising results on the HotpotQA, TriviaQA, NQ-WC datasets.

Critical strengths
  • The proposed approach is simple but somewhat incremental improvement over LOS-Net and shows promising results.

  • The proposed approach is simile to train.

  • The paper is well written and easy to understand.

Critical weaknesses
  • The evaluation is somewhat limited. The paper considers only the older Gemma-4-12B-IT, Llama-3.2-3B-Instruct, and Ministral-3-8B-Instruct which are far from being state of the art currently. The paper should consider newer state of the art models such as Qwen3.5/3.6 family (2B/4B or even 9B).

  • The evaluation is also limited to free datasets: HotpotQA, TriviaQA, NQ-WC. The paper should consider additional reasoning datasets such as GSM8k.

  • The proposed approach is not training free. The GURI detector is LLM specific and needs to trained for any new LLM and for every dataset.

  • The paper does not compare to prior works such as "Enhancing hallucination detection through noise injection, ICLR 2026" which is a training free approach.

  • This grey box approach is also prone to errors being introduced by the GURI detector. The paper should analyze any systematic biases introduced by training the GURI detector.

  • The proposed GURI approach only considers the answer tokens in it's hallucination detection. reasoning chains are not considered, which may be very helpful for hallucination detection.

Questions for authors
  • The paper should analyze the effect of the training set on GURI detector accuracy.
  • The paper should consider additional datasets such as GSM8k which requires reasoning.
  • The paper should consider prior training free approaches as baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.