- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Token-FAR, a technique for detecting hallucinations that is based on the local geometry of the internal token representations. Instead of training a parametric classifier to identify hallucinations or generating several possible responses during inference, Token-FAR creates an offline factual reference dictionary by using hidden representations taken from known factual answers. For each content token in a candidate response, the method finds its nearest factual token representations and reconstructs the candidate representation with a local affine ridge model. The residual obtained from this reconstruction is taken to be a measure of the token's deviation from the locally supported factual geometry, and the token-level residuals are averaged in order to obtain an answer-level hallucination score.
The paper also gives a geometric breakdown of the ridge reconstruction residual into an orthogonal part which measures the displacement outside the local factual subspace and a component within the subspace caused by ridge shrinkage. The authors claim that hallucinated tokens should have larger reconstruction residuals since their hidden representations are less well supported by the factual neighborhoods.
Experiments were carried out on the TruthfulQA, HaluEval-QA, FELM-Science, and FaithDial datasets using the Qwen2.5-7B-Instruct and Llama-3.2-3B-Instruct models. Token-FAR was compared with nine training-free baseline methods and achieved the highest reported AUROC in seven out of the eight model–dataset combinations. Furthermore, the paper includes ablation studies on affine as compared to linear reconstruction, on the quality of the factual reference, on the decoder layer, on the neighborhood size, on the ridge coefficient, and on the size of the reference set.
The underlying concept is straightforward and may offer practical value. However, I am concerned about the framing of the supervision setting, the fairness and consistency of the baseline evaluation, and whether the current experiments sufficiently demonstrate that the proposed geometric signal captures factuality rather than merely reflecting similarity to the domain or reference distribution.
Critical strengths
The method does not require fitting a parametric detector to hallucinated and factual labels. Instead, it uses a factual reference set to compute a deterministic local geometric score that quantifies the extent to which each candidate token is supported by nearby factual representations. This approach is straightforward and offers a clear interpretation in terms of support on the local manifold.
The residual decomposition further strengthens the interpretability of the method. In the HaluEval-QA analysis, the increased residual for hallucinated responses is reported to arise mainly from the orthogonal component rather than ridge shrinkage, which is consistent with the proposed geometric interpretation.
The paper presents several informative ablation studies derived from the geometric construction. In Table 2 the authors compare nearest-neighbor dissimilarity, local linear ridge reconstruction, and local affine ridge reconstruction, with the affine formulation being consistently the best.
The study also examines the effect of dictionary corruption, the choice of decoder layer, neighborhood size, ridge strength, and the size of the factual reference set. Specifically, the analysis of the reference set size indicates that performance on HaluEval-QA levels off rather quickly, since 25 factual examples are already nearly as good as the result obtained with 300 examples. This provides useful evidence that the method may not need an extremely large reference set.
Critical weaknesses
- The idea that the process is 'training-free' fails to reflect the extent of the supervision and prior factual knowledge needed by Token-FAR.
Token-FAR does not involve training a parametric binary detector, but it is not a resource-free or supervision-free approach. Rather, it relies on a carefully curated set of known factual answers from the target domain, which are processed through the same model to construct the factual representation dictionary.
- Relying on having access to a group of reliable factual examples from the target benchmark/domain.
Table 3 supports this worry: a significant decline in performance is observed when the factual dictionary is substituted with a generated-response or mixed dictionary.
I recommend that the paper employ matched-reference baselines, such as cosine or kNN distance, Mahalanobis distance, PCA or local-subspace residual, LOF, or another one-class detector utilizing the same factual reference representations. This would help determine whether the observed improvement is attributable to the proposed affine reconstruction or simply to access to positive factual examples.
- There is an apparent inconsistency in the scoring direction in the baseline evaluation, which could cause certain comparisons to be unfair.
The appendix, A.2, of the paper states that higher scores indicate hallucination. However, the case-study section makes it clear that in the case of LexicalSimilarity a higher score means non-hallucination. This is especially worrying since LexicalSimilarity achieves an AUROC of 15.3 on Qwen/HaluEval-QA. An AUROC much below 0.5 may be a sign that a useful signal has simply been assessed with the opposite orientation. In the same scenario, Table 1 also shows an unusually low EigenScore AUROC of 9.2.
The paper should specify the direction convention for each baseline and clarify that AUROC was calculated using each method's intended hallucination score. Without such clarification, the current baseline table may not provide a fair comparison, particularly if LexicalSimilarity was evaluated without score inversion.
-
The method should be described as 'single-generation' rather than 'low-cost' unless empirical evidence regarding runtime is provided.
-
Token-FAR is designed as a detector at the token level, and Figures 2, 3, and 5 show the token residuals. Yet the main metric used is AUROC at the answer level, and the analysis mainly compares 'tokens from hallucinated responses' with 'tokens from factual responses'.
-
The factual dictionary is constructed independently for each target dataset. As a result, a low reconstruction residual indicates that the candidate representation is close to the distribution of actual answers from the same benchmark, rather than being factual in a general sense. The paper does not report experiments involving cross-dataset evaluation, such as building the factual dictionary from one factuality benchmark and applying it unchanged to another. Such an experiment would be valuable for determining whether the learned local geometry reflects transferable factuality or merely captures dataset-specific lexical and semantic regularities.
Questions for authors
- In what exact way were the score instructions dealt with when calculating the AUROC for each baseline?
Appendix A.2 says that higher scores always mean hallucination, whereas Appendix C says that greater LexicalSimilarity scores indicate non-hallucination. Was the LexicalSimilarity score negated before the computation of Table 1? In the event that it wasn't, the HaluEval AUROC of 15.3 might be orientation-reversed. Please tell us the exact score transformation that was used for each baseline and rerun the comparison if needed.
-
To what extent does the improvement of Token-FAR result from having access to factual references within the target domain rather than from the affine reconstruction itself?
-
Are the authors able to assess reference transfer between datasets?
For instance, you could build the factual dictionary using TruthfulQA and then evaluate it on HaluEval/FELM without making any changes, or alternatively, start by evaluating on HaluEval/FELM and then use TruthfulQA. If performance drops significantly in this case, it would indicate that the method is mainly assessing support for the target domain rather than general factuality.
- Are the authors able to provide the real inference latency and memory usage?
The method involves a single response but for each content token it needs to carry out a nearest-neighbor retrieval over a possibly large hidden-state dictionary. Please provide a report on its runtime and the amount of reference-dictionary memory when compared with those of Perplexity, P(True), NCI/fDBD, and the sampling-based approaches.
6 comments
Response to Weakness 3 and Question 1
W3 (score direction) and Q1. Specify the score transformation for each baseline, verify whether Lexical Similarity was inverted before AUROC computation, and explain the unusually low Lexical Similarity and EigenScore AUROCs on Qwen/HaluEval-QA.
We use hallucination as the positive class and specify the direction of each candidate score before evaluation. We used before computing the Lexical Similarity AUROC in Table 1. The case study describes the raw similarity convention, where higher similarity favors a factual prediction. The AUROC uses its complement.
| Baseline | Score used for AUROC |
|---|---|
| Perplexity | Exponential of mean token negative log-likelihood |
| LN-Entropy | Sum of token predictive entropies divided by the square root of the token count |
| Energy | Negative mean of the tokenwise log-sum-exp of logits |
| P(True) | |
| Lexical Similarity | minus mean ROUGE-L between the candidate and the support responses |
| Semantic Entropy | Surprisal of the candidate's semantic cluster |
| Candidate-augmented EigenScore | , with the support set fixed for question |
| NCI | Negative mean proximity score |
| fDBD | Negative decision-boundary distance |
-
Fixed-candidate score check. We recomputed Lexical Similarity and the EigenScore difference on the 1,000 fixed Qwen/HaluEval-QA candidates with the same support generations and saved EigenScore embeddings. The recomputation recovered the Table 1 AUROCs of 15.3% and 9.2%, respectively. The low values reflect the rankings produced by the specified scores on this pool. The score directions were fixed before evaluation.
-
Candidate-level versus question-level EigenScore. The candidate-augmented score measures the change in the covariance statistic when an answer is added to a fixed support set. A smaller increment can occur when the candidate embedding adds little variation beyond that already represented by the support embeddings. On Qwen/HaluEval-QA, the support-only statistic yields 52.8% AUROC, while the candidate increment yields 9.2%, locating the reversal in the candidate-level adaptation. We also examined the saved Llama scores. Factual and hallucinated answers had median lengths of 2 and 9 words, respectively. On the same 690 answers at lengths represented in both classes, AUROC was 14.7% when pooled and 46.8% when only equal-length factual–hallucinated pairs were compared. This diagnostic links the Llama reversal to answer length or correlated differences in response format.
Response to Weakness 4
W4. The method assigns token scores, while the main evaluation uses answer-level AUROC and the analysis groups tokens by their parent answer's label. Evaluate localization using actual span labels.
Table 1 evaluates one mean-pooled score per answer. The token visualizations show how scores vary within answers, and answer labels leave the error locations unspecified. We added two controlled evaluations using known spans.
-
Answer-level sensitivity to a single edit. We construct 34 HaluEval-QA pairs with the same question, source context, and word count, altering one factual span. Token-FAR gives the false answer a higher score in 33/34 pairs for Qwen and 34/34 for Llama. This measures whether the answer score responds to a localized factual change.
-
Within-passage localization. We separately test 24 matched HaluEval-QA passage pairs of 30–109 words with one factual span edited. We use the edited span to label the target tokens and compare Token-FAR against LOF and the pure local orthogonal residual using the same factual references and neighbors. Token-FAR achieves the highest Top-1 and AP for both models in this controlled comparison (Table R3). Compared with LOF, its localization AP is 8.8 points higher for Qwen and 12.9 higher for Llama. Compared with the orthogonal residual, the gains are 10.0 and 30.0 points, respectively.
Table R3. Within-passage error localization on the 24 edited HaluEval-QA passages. Each entry reports Top-1 / AP (%), averaged across passages.
| Model | Token-FAR | LOF | Local orthogonal residual |
|---|---|---|---|
| Qwen | 16.7 / 19.3 | 8.3 / 10.5 | 4.2 / 9.3 |
| Llama | 45.8 / 51.7 | 33.3 / 38.8 | 12.5 / 21.7 |
- Native segment annotations. We also align native FELM-Science annotations to scored tokens. On the 20 answers containing both aligned factual and false segments, mean within-answer segment AUROC is 60.2% for Qwen and 49.5% for Llama.
Response to Weaknesses 1 and 2 and Question 2
Thank you for the careful review and for highlighting the interpretability of the residual decomposition.
Reference supervision. Token-FAR uses a frozen LLM and requires trusted factual answers from a designated reference set. We encode those answers offline and build a separate dictionary for each model and dataset. The term “training-free” refers specifically to the absence of a parametric detector fitted on hallucination labels. Factual supervision enters through the dictionary. Evaluation candidates are used as queries, and their labels are used for evaluation.
Reference quality and affine centering. Tables 2 and 3 in the original submission ablate the scoring rule and dictionary composition, respectively. In the new control, we replace factual answers with incorrect answers to the same reference prompts, match reference-token counts, and fix the target candidates and scorer. We also change the centering step with the factual dictionary held fixed. Factual references increase AUROC by 23.8–48.8 points on HaluEval-QA and FaithDial and by 2.9–5.0 points on TruthfulQA. Centering adds 0.2–1.4 points in these six settings (Table R1). Reference quality has the larger effect in this comparison.
Table R1. Effects of factual references and affine centering. Values are AUROC percentage-point differences with paired 95% intervals.
Table R2. Answer-level AUROC (%) using the same factual dictionary.