acceptodds
Under review as a conference paper at ICLR 2027

Token-FAR: Hallucination Detection from a Token Local Geometry Perspective

Abstract

Hallucination remains a major obstacle to the reliable deployment of Large Language Models (LLMs). However, existing hallucination detection methods often rely on repeated generation, semantic comparison, or trained detectors, causing extra inference cost or requiring supervision. In this paper, we investigate whether the local geometry of the internal token representations from a single candidate response can provide an effective hallucination signal without the need for training. To this end, we propose Token-FAR (Token-level Factual Affine Reconstruction), a token-level geometric hallucination detector based on local affine reconstruction with respect to a factual reference dictionary. To be specific, in Token-FAR, we utilize the hidden states of factual answers to construct a factual reference dictionary, and then for each candidate content token, we compute an affine ridge reconstruction residual with respect to a local factual neighborhood. We observe that the content tokens in a candidate answer that are poorly reconstructed by the local factual neighborhood will result in larger residuals, which provide a signal to indicate an occurrence of hallucination. By aggregating the token-level residuals, we define a score for detecting answer-level hallucination. Our proposed Token-FAR requires only a single candidate response at inference time and does not need to train a parametric detector using hallucination labels. We conduct extensive experiments and ablation studies on four factuality benchmark datasets across two LLMs. Experimental results demonstrate the effectiveness of our proposed approach, showing that local affine reconstruction with respect to a factual reference dictionary can provide an effective geometric signal for hallucination detection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hallucination detection, large language models, affine reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_fhQLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Token-FAR, a technique for detecting hallucinations that is based on the local geometry of the internal token representations. Instead of training a parametric classifier to identify hallucinations or generating several possible responses during inference, Token-FAR creates an offline factual reference dictionary by using hidden representations taken from known factual answers. For each content token in a candidate response, the method finds its kk nearest factual token representations and reconstructs the candidate representation with a local affine ridge model. The residual obtained from this reconstruction is taken to be a measure of the token's deviation from the locally supported factual geometry, and the token-level residuals are averaged in order to obtain an answer-level hallucination score.

The paper also gives a geometric breakdown of the ridge reconstruction residual into an orthogonal part which measures the displacement outside the local factual subspace and a component within the subspace caused by ridge shrinkage. The authors claim that hallucinated tokens should have larger reconstruction residuals since their hidden representations are less well supported by the factual neighborhoods.

Experiments were carried out on the TruthfulQA, HaluEval-QA, FELM-Science, and FaithDial datasets using the Qwen2.5-7B-Instruct and Llama-3.2-3B-Instruct models. Token-FAR was compared with nine training-free baseline methods and achieved the highest reported AUROC in seven out of the eight model–dataset combinations. Furthermore, the paper includes ablation studies on affine as compared to linear reconstruction, on the quality of the factual reference, on the decoder layer, on the neighborhood size, on the ridge coefficient, and on the size of the reference set.

The underlying concept is straightforward and may offer practical value. However, I am concerned about the framing of the supervision setting, the fairness and consistency of the baseline evaluation, and whether the current experiments sufficiently demonstrate that the proposed geometric signal captures factuality rather than merely reflecting similarity to the domain or reference distribution.

Critical strengths

The method does not require fitting a parametric detector to hallucinated and factual labels. Instead, it uses a factual reference set to compute a deterministic local geometric score that quantifies the extent to which each candidate token is supported by nearby factual representations. This approach is straightforward and offers a clear interpretation in terms of support on the local manifold.

The residual decomposition further strengthens the interpretability of the method. In the HaluEval-QA analysis, the increased residual for hallucinated responses is reported to arise mainly from the orthogonal component rather than ridge shrinkage, which is consistent with the proposed geometric interpretation.

The paper presents several informative ablation studies derived from the geometric construction. In Table 2 the authors compare nearest-neighbor dissimilarity, local linear ridge reconstruction, and local affine ridge reconstruction, with the affine formulation being consistently the best.

The study also examines the effect of dictionary corruption, the choice of decoder layer, neighborhood size, ridge strength, and the size of the factual reference set. Specifically, the analysis of the reference set size indicates that performance on HaluEval-QA levels off rather quickly, since 25 factual examples are already nearly as good as the result obtained with 300 examples. This provides useful evidence that the method may not need an extremely large reference set.

Critical weaknesses
  1. The idea that the process is 'training-free' fails to reflect the extent of the supervision and prior factual knowledge needed by Token-FAR.

Token-FAR does not involve training a parametric binary detector, but it is not a resource-free or supervision-free approach. Rather, it relies on a carefully curated set of known factual answers from the target domain, which are processed through the same model to construct the factual representation dictionary.

  1. Relying on having access to a group of reliable factual examples from the target benchmark/domain.

Table 3 supports this worry: a significant decline in performance is observed when the factual dictionary is substituted with a generated-response or mixed dictionary.

I recommend that the paper employ matched-reference baselines, such as cosine or kNN distance, Mahalanobis distance, PCA or local-subspace residual, LOF, or another one-class detector utilizing the same factual reference representations. This would help determine whether the observed improvement is attributable to the proposed affine reconstruction or simply to access to positive factual examples.

  1. There is an apparent inconsistency in the scoring direction in the baseline evaluation, which could cause certain comparisons to be unfair.

The appendix, A.2, of the paper states that higher scores indicate hallucination. However, the case-study section makes it clear that in the case of LexicalSimilarity a higher score means non-hallucination. This is especially worrying since LexicalSimilarity achieves an AUROC of 15.3 on Qwen/HaluEval-QA. An AUROC much below 0.5 may be a sign that a useful signal has simply been assessed with the opposite orientation. In the same scenario, Table 1 also shows an unusually low EigenScore AUROC of 9.2.

The paper should specify the direction convention for each baseline and clarify that AUROC was calculated using each method's intended hallucination score. Without such clarification, the current baseline table may not provide a fair comparison, particularly if LexicalSimilarity was evaluated without score inversion.

  1. The method should be described as 'single-generation' rather than 'low-cost' unless empirical evidence regarding runtime is provided.

  2. Token-FAR is designed as a detector at the token level, and Figures 2, 3, and 5 show the token residuals. Yet the main metric used is AUROC at the answer level, and the analysis mainly compares 'tokens from hallucinated responses' with 'tokens from factual responses'.

  3. The factual dictionary is constructed independently for each target dataset. As a result, a low reconstruction residual indicates that the candidate representation is close to the distribution of actual answers from the same benchmark, rather than being factual in a general sense. The paper does not report experiments involving cross-dataset evaluation, such as building the factual dictionary from one factuality benchmark and applying it unchanged to another. Such an experiment would be valuable for determining whether the learned local geometry reflects transferable factuality or merely captures dataset-specific lexical and semantic regularities.

Questions for authors
  1. In what exact way were the score instructions dealt with when calculating the AUROC for each baseline?

Appendix A.2 says that higher scores always mean hallucination, whereas Appendix C says that greater LexicalSimilarity scores indicate non-hallucination. Was the LexicalSimilarity score negated before the computation of Table 1? In the event that it wasn't, the HaluEval AUROC of 15.3 might be orientation-reversed. Please tell us the exact score transformation that was used for each baseline and rerun the comparison if needed.

  1. To what extent does the improvement of Token-FAR result from having access to factual references within the target domain rather than from the affine reconstruction itself?

  2. Are the authors able to assess reference transfer between datasets?

For instance, you could build the factual dictionary using TruthfulQA and then evaluate it on HaluEval/FELM without making any changes, or alternatively, start by evaluating on HaluEval/FELM and then use TruthfulQA. If performance drops significantly in this case, it would indicate that the method is mainly assessing support for the target domain rather than general factuality.

  1. Are the authors able to provide the real inference latency and memory usage?

The method involves a single response but for each content token it needs to carry out a nearest-neighbor retrieval over a possibly large hidden-state dictionary. Please provide a report on its runtime and the amount of reference-dictionary memory when compared with those of Perplexity, P(True), NCI/fDBD, and the sampling-based approaches.

6 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 and 2 and Question 2

W1, W2, and Q2. Clarify the factual supervision required by Token-FAR and isolate the contribution of affine reconstruction using baselines with access to the same references.

Thank you for the careful review and for highlighting the interpretability of the residual decomposition.

  • Reference supervision. Token-FAR uses a frozen LLM and requires trusted factual answers from a designated reference set. We encode those answers offline and build a separate dictionary for each model and dataset. The term “training-free” refers specifically to the absence of a parametric detector fitted on hallucination labels. Factual supervision enters through the dictionary. Evaluation candidates are used as queries, and their labels are used for evaluation.

  • Reference quality and affine centering. Tables 2 and 3 in the original submission ablate the scoring rule and dictionary composition, respectively. In the new control, we replace factual answers with incorrect answers to the same reference prompts, match reference-token counts, and fix the target candidates and scorer. We also change the centering step with the factual dictionary held fixed. Factual references increase AUROC by 23.8–48.8 points on HaluEval-QA and FaithDial and by 2.9–5.0 points on TruthfulQA. Centering adds 0.2–1.4 points in these six settings (Table R1). Reference quality has the larger effect in this comparison.

Table R1. Effects of factual references and affine centering. Values are AUROC percentage-point differences with paired 95% intervals.

DatasetModelReference tokensFactual − hallucinated referencesCentered − uncentered ridge
HaluEval-QAQwen1,092+46.6 [43.5, 49.9]+1.4 [1.2, 1.7]
HaluEval-QALlama999+48.8 [45.3, 52.4]+0.2 [0.1, 0.4]
FaithDialQwen5,914+23.8 [22.3, 25.4]+0.2 [0.1, 0.2]
FaithDialLlama5,764+26.3 [24.7, 28.0]+0.2 [0.2, 0.3]
TruthfulQAQwen2,392+2.9 [0.0, 6.0]+0.9 [0.5, 1.2]
TruthfulQALlama2,374+5.0 [1.9, 7.7]+0.9 [0.6, 1.3]
  • Baselines using the same factual dictionary. We compare Token-FAR with LOF, diagonal Mahalanobis distance, global PCA residual, and cosine 10-NN. Token-FAR leads in five of eight model–dataset settings. LOF leads on HaluEval-QA for both models, and global PCA leads for Llama on FELM-Science. The shared-reference comparison supports a benefit from affine scoring in several settings and shows where other geometric scores work better.

Table R2. Answer-level AUROC (%) using the same factual dictionary.

DatasetModelToken-FARLOFDiag. MahalanobisGlobal PCACosine 10-NN
TruthfulQAQwen62.159.855.152.659.3
TruthfulQALlama70.668.258.355.763.4
HaluEval-QAQwen91.794.191.387.681.7
HaluEval-QALlama97.197.996.795.393.5
FELM-ScienceQwen62.757.744.760.560.1
FELM-ScienceLlama70.569.857.874.270.5
FaithDialQwen80.574.764.176.379.6
FaithDialLlama78.276.054.473.777.8
AuthorsOpenReview ↗

Response to Weakness 3 and Question 1

W3 (score direction) and Q1. Specify the score transformation for each baseline, verify whether Lexical Similarity was inverted before AUROC computation, and explain the unusually low Lexical Similarity and EigenScore AUROCs on Qwen/HaluEval-QA.

We use hallucination as the positive class and specify the direction of each candidate score before evaluation. We used 1−ROUGE-L‾1-\overline{\mathrm{ROUGE\text{-}L}} before computing the Lexical Similarity AUROC in Table 1. The case study describes the raw similarity convention, where higher similarity favors a factual prediction. The AUROC uses its complement.

BaselineScore used for AUROC
PerplexityExponential of mean token negative log-likelihood
LN-EntropySum of token predictive entropies divided by the square root of the token count
EnergyNegative mean of the tokenwise log-sum-exp of logits
P(True)1−P(True)1-P(\mathrm{True})
Lexical Similarity11 minus mean ROUGE-L between the candidate and the support responses
Semantic EntropySurprisal of the candidate's semantic cluster
Candidate-augmented EigenScoreE(Sq∪{a})−E(Sq)E(S_q\cup\{a\})-E(S_q), with the support set SqS_q fixed for question qq
NCINegative mean proximity score
fDBDNegative decision-boundary distance
  • Fixed-candidate score check. We recomputed Lexical Similarity and the EigenScore difference on the 1,000 fixed Qwen/HaluEval-QA candidates with the same support generations and saved EigenScore embeddings. The recomputation recovered the Table 1 AUROCs of 15.3% and 9.2%, respectively. The low values reflect the rankings produced by the specified scores on this pool. The score directions were fixed before evaluation.

  • Candidate-level versus question-level EigenScore. The candidate-augmented score measures the change in the covariance statistic when an answer is added to a fixed support set. A smaller increment can occur when the candidate embedding adds little variation beyond that already represented by the support embeddings. On Qwen/HaluEval-QA, the support-only statistic yields 52.8% AUROC, while the candidate increment yields 9.2%, locating the reversal in the candidate-level adaptation. We also examined the saved Llama scores. Factual and hallucinated answers had median lengths of 2 and 9 words, respectively. On the same 690 answers at lengths represented in both classes, AUROC was 14.7% when pooled and 46.8% when only equal-length factual–hallucinated pairs were compared. This diagnostic links the Llama reversal to answer length or correlated differences in response format.

AuthorsOpenReview ↗

Response to Weakness 4

W4. The method assigns token scores, while the main evaluation uses answer-level AUROC and the analysis groups tokens by their parent answer's label. Evaluate localization using actual span labels.

Table 1 evaluates one mean-pooled score per answer. The token visualizations show how scores vary within answers, and answer labels leave the error locations unspecified. We added two controlled evaluations using known spans.

  • Answer-level sensitivity to a single edit. We construct 34 HaluEval-QA pairs with the same question, source context, and word count, altering one factual span. Token-FAR gives the false answer a higher score in 33/34 pairs for Qwen and 34/34 for Llama. This measures whether the answer score responds to a localized factual change.

  • Within-passage localization. We separately test 24 matched HaluEval-QA passage pairs of 30–109 words with one factual span edited. We use the edited span to label the target tokens and compare Token-FAR against LOF and the pure local orthogonal residual using the same factual references and neighbors. Token-FAR achieves the highest Top-1 and AP for both models in this controlled comparison (Table R3). Compared with LOF, its localization AP is 8.8 points higher for Qwen and 12.9 higher for Llama. Compared with the orthogonal residual, the gains are 10.0 and 30.0 points, respectively.

Table R3. Within-passage error localization on the 24 edited HaluEval-QA passages. Each entry reports Top-1 / AP (%), averaged across passages.

ModelToken-FARLOFLocal orthogonal residual
Qwen16.7 / 19.38.3 / 10.54.2 / 9.3
Llama45.8 / 51.733.3 / 38.812.5 / 21.7
  • Native segment annotations. We also align native FELM-Science annotations to scored tokens. On the 20 answers containing both aligned factual and false segments, mean within-answer segment AUROC is 60.2% for Qwen and 49.5% for Llama.
Reviewer_Udi9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces Token-FAR, a white-box methodology for hallucination detection in LLM responses, based on classifying the responses with respect to a dictionary of pre-learned factual references (seemingly drawn from a disjoint split of the very same dataset later used for testing). The method requires access to the internal representations of the LLM at a fixed middle layer ℓmid\ell_\text{mid}. Responses are labelled through ROUGE-L > 0.7 or semantic similarity > 0.5 (Section 4.1). For each content token of the target response, Token-FAR selects the kk closest reference token representations in the dictionary and builds the local affine subspace they span, against which the token is evaluated; this determines a residual "distance", which is averaged across the content tokens of the response to create a hallucination score. The performance of Token-FAR is evaluated with AUC-ROC only again a labelling evaluated by ROUGE-L > 0.7 or semantic similarity > 0.5 (Section 4.1). No actual threshold for the classification is provided (Appendix C.1 only reports thresholds for a few case studies), hence no point statistics (accuracy, confusion, etc) is evaluated. The results are reported on four literature benchmarks (TruthfulQA, HaluEval-QA, FELM-Science, FaithDial), across two small-sized instruction-tuned LLMs (Qwen2.5-7B and Llama-3.2-3B), and against nine literature baselines. Further analyses of the reconstruction geometry are provided, including the local factual geometry and the residual distributions, alongside ablations on the residual definition and on the dictionary construction, a layer-sensitivity study, and case studies.

Critical strengths

The reviewer appreciates the structure of the paper and its intuition. The main strengths of the paper lie in the ideas it brings to the field of hallucination detection: the idea is interesting, the results are promising, and the work has, in the opinion of the reviewer, considerable potential.

  1. The parallel/perpendicular decomposition of the residual is, to the best of the reviewer's knowledge, new in hallucination detection. Splitting the residual into its components orthogonal and parallel to the local factual subspace (δ⊥2\delta_\perp^2 and δ∥2\delta_\parallel^2, Eq. (9)) is customary in geometry, yet largely unexplored in hallucination detection.
  2. The online cost is extremely limited, which favours deployability. Token-FAR requires a single candidate response at inference time (Section 3.5). Moreover, the process being presented as label-free should ease deployment and make the method less dependent on the benchmark used for evaluation. However, after a careful reading, the label-free property does not hold as stated (see Critical Weakness 2).

Unfortunatelly, these claims are currently not substantiated in a convincing manner (see below).

Critical weaknesses

It is the opinion of the reviewer that the experimental setting is immature and deserves a more careful analysis before the work can be considered a full publication.

  1. The evaluation protocol does not allow a cautious assessment of the results. Indeed, several aspects of the experimental setting jointly prevent the claims from being verified:
    • Single metric. AUC-ROC is the sole metric used (Section 4.1), which is not sufficient to fully analyse a classifier. AUC-ROC can be overly optimistic and less reliable under class imbalance, and the hallucination rates are not reported. Moreover, AUC-ROC alone does not allow the protocol to be instantiated: even an extremely high AUC-ROC only implies that an optimal threshold with high precision and recall exists, and such a threshold might still be unstable and dataset-dependent. Thresholds are selected only to illustrate the case studies of Appendix C.1 (HaluEval-QA with Llama), which, in the view of the reviewer, does not amount to a proper study of the operating threshold. As a consequence, none of the above can be evaluated for the main results, and no claim on the transferability of the threshold from one dataset to another can be made. The reviewer regards this as a major flaw; at the very least, confusion matrices and per-dataset results should be reported.
    • Labelling reliability. Judging responses by ROUGE-L > 0.7 or semantic similarity > 0.5 (Section 4.1) is not a reliable way to detect hallucinations, particularly if the labelled set is not then analysed by humans (at least partially) or compared with a different methodology, such as LLM-as-a-judge. This tagging might in fact smooth the separation between hallucinated and genuine responses (see Question Q4 and Suggestion S1).
    • Unsubstantiated parameter choices. The choices of ℓmid\ell_\text{mid}, RR, kk, and λ\lambda are not adequately substantiated (see Questions Q6 and Q7).
    • Limited model coverage. Only two small LLMs are adopted throughout, which might yield more predictable outcomes and less discriminative power (see Suggestion S3).
    • Missing dataset statistics. Apart from their size, no statistics are presented on the responses, most notably the hallucination rate, but also the topic distribution or other statistics useful to understand the results.
    • No code. Many of these points cannot be settled from the manuscript alone, as the code is not included in the submission.
  2. The "training-free" claim appears to be a major overclaim. It is not clear whether the factual reference dictionary uses the references of all datasets together, or only those drawn from the dataset under study; the text (Appendix A.1–A.2, Table 4) suggests the latter. If this is the case, presenting the method as training-free (Abstract; Section 3.5) is a major overclaim, as labelled samples must still be provided, and they must be in-domain with respect to the evaluated answers. Moreover, the teacher forcing of the labelled factual answers through the evaluated LLM, required to build the dictionary (Section 3.2), is itself part of what makes the method not training-free. Even under the narrower definition of Section 4.2 (not fitting a parametric detector on labelled hallucination data), the labelled samples are, at the very least, needed to learn the decision threshold. A study of cross-dataset generalisation, i.e., building the dictionary on one dataset and evaluating on the questions of another, would lower the impact of this requirement (see Suggestion S5). Likewise, since the dictionary consists of the internal representations of a specific LLM, its transfer from one LLM to another is probably impossible, and at the very least unstudied. Without such analyses, the contribution remains marginal.
Questions for authors

Clarification requests

  • Q1 (decision rule). Eq. (10) aggregates the token residuals δ(z)\delta(\boldsymbol{z}) of Eq. (7) into the answer-level score S(X,Y)S(\mathcal{X}, \mathcal{Y}); however, it is not clear how this score is ultimately adopted to declare a response hallucinated or genuine. Beyond the thresholds selected for the case studies of Appendix C.1, is a specific threshold chosen for the main results and, if so, is it the same across datasets? Please see also Critical Weakness 1 on the choice of the metrics.
  • Q2 (reference set). How is the reference set of RR questions built? Appendix A.1 only partially answers this, as the overall procedure remains unclear. In particular, is the dictionary used for each dataset built only from the references of that dataset, or are the references of all datasets used together?
  • Q3 (factual tokens and teacher forcing). How is the set Ym⋆\mathcal{Y}^\star_m derived from Ym\mathcal{Y}_m, i.e., how are the factual tokens filtered out from all the tokens of the responses to Xm\mathcal{X}_m (Section 3.1)? What exactly is meant by "teacher forcing" when extracting the normalised representations \boldsymbol{z}^\star_{m,j} (Section 3.2), i.e., which ground truth is substituted at each step?
  • Q4 (labelling protocol). Section 4.1 states that generated responses are labelled as factual when their maximum ROUGE-L exceeds 0.7 or their maximum semantic similarity exceeds 0.5. Conversely, Appendix A.1 and Table 4 state that, for HaluEval-QA, FELM-Science, and FaithDial, candidate answers and binary labels are provided by the benchmark, the rule above applying to TruthfulQA only. To the best of the reviewer's understanding, the reference set is partially generated and then tagged (TruthfulQA), and partially obtained by teacher forcing with labels taken from the dataset. In turn, the analysed set on which AUC-ROC is computed must in all cases be constructed under the two LLMs, since the method needs their internal representations at layer ℓ\ell to work. Please confirm this reading and reconcile the two descriptions, clarifying for each dataset how the ground truth of the responses is obtained.
  • Q5 (FELM-Science sizes). Table 4 reports 1,000 evaluation responses for FELM-Science, whereas, to the best of the reviewer's knowledge, the Science domain of FELM comprises 125 responses (Chen et al., 2023, Table 2; arXiv:2310.00741). Please clarify how the evaluation set was obtained and why only 50 reference answers are used, against 300 for the other datasets. Should the 1,000 evaluation responses be confirmed independent, this would not seem to be a size issue, and it should be carefully explained so as not to suggest cherry-picking; is it due to the requirement that all segments be annotated as factual (Appendix A.1)?
  • Q6 (layer choice). Why specifically ℓmid=⌈L/2⌉\ell_\text{mid} = \lceil L/2 \rceil? The value is studied in "Layer sensitivity" (Section 4.3, Figure 4); however, why not span all the layers, and why start from layer 7? Moreover, having tested the L/2L/2 heuristic on only two models, both with 28 layers, offers little support.
  • Q7 (choice of RR, kk, and λ\lambda). How were RR, k=10k = 10, and λ=0.01\lambda = 0.01 chosen? Appendix D answers this in part; however, the implications of Figures 6–8 should be discussed in more depth. In particular, why is the sensitivity to kk and λ\lambda so limited for all datasets but FELM-Science, and why does the performance decrease at R=300R = 300? Why is the study on RR conducted on one dataset and one LLM only, whilst the studies on kk and λ\lambda cover all eight combinations? Finally, why is R=300R = 300 used in the main experiments (Table 4), although it performs worse than R=200R = 200?

Suggestions

  • S1 (independent validation of the labels). The reviewer suggests validating the labels against a different methodology, e.g., an LLM-as-a-judge approach, and/or a partial human analysis. Evaluating an LLM-as-a-judge directly on the responses would also provide an additional performance row for a thresholded results table (see S2), which would be an interesting result per se, as it may suggest coherence or incoherence in the overall methodology:
    • an extremely high judge accuracy would suggest that the tagging methodology is sound;
    • a lower accuracy might imply either an inaccurate judge or a misleading tagging; in the latter case, an accuracy lower than that of Token-FAR would suggest that Token-FAR may capture cues shared with the ROUGE-L and semantic-similarity tagging.
  • S2 (thresholded evaluation). Alongside AUC-ROC, a confusion matrix and per-dataset results at a fixed, representative threshold should be reported, together with an assessment of whether that threshold transfers across datasets (see Critical Weakness 1).
  • S3 (larger models). It is advisable to include larger LLMs in the analysis, as the two small models adopted may yield more predictable outcomes and less discriminative power.
  • S4 (repeated-response test). An experiment which, in the view of the reviewer, is missing is a repeated-response test: how do hallucinated and genuine responses to the same prompt, obtained by repeated sampling at non-zero temperature, behave?
  • S5 (cross-dataset and cross-model generalisation). Transferring the dictionary built on one dataset to the questions of another would lower the impact of the in-domain reference requirement (see Critical Weakness 2). The transfer of the dictionary from one LLM to another, although probably impossible, should at least be studied.
  • S6 (dataset statistics). Reporting the hallucination rate, the topic distribution, and other statistics of the generated responses would help to understand the results.
  • S7 (code). Many of the above questions would be settled if the code were included; the reviewer therefore strongly advises adding the actual code and experimental setup to the submission.
  • S8 (time benchmark, secondary). Although secondary, a time benchmark substantiating the limited online cost (Section 3.5), potentially compared with that of the baselines, should be reported.

Minor points

  • The paper is generally well written, but it requires non-negligible effort, particularly in the mathematical part.
  • From Eq. (3) onwards, why is the subscript zz (e.g., in μz\boldsymbol{\mu}_z and Dz\boldsymbol{D}_z) not bold?
  • By col(A)\mathrm{col}(\boldsymbol{A}), with A\boldsymbol{A} a matrix, do the authors refer to its columns? Please define for completeness.
  • In "Comparison on residual distributions" (Section 4.3), Llama-3.2-3B is used instead of the instruction-tuned variant stated in "Local factual geometry validation"; the reviewer assumes this is only a typo, but please confirm.
  • Since only two models are used, space could be saved by referring to them by family name once they have been introduced.
13 comments
AuthorsOpenReview ↗

Response to Weakness 1, Question 1, and Suggestions 2 and 6

W1, Q1, S2, S6. Explain how the answer score becomes a decision, and report class balance, fixed-threshold performance, and threshold transfer.

Thank you for your constructive suggestions!

Fixed-threshold evaluation

We predict hallucination when the answer-level score in Eq. (10) satisfies S≥τ∗S\geq\tau^{\ast}. We assign approximately 20% of the groups to calibration, grouping by normalized prompt where applicable and by dialogue for FaithDial. Within each model–dataset setting, all methods use the same split. For each method, model, and dataset, we select τ∗\tau^{\ast} by maximizing TPR⋅TNR\sqrt{\mathrm{TPR}\cdot\mathrm{TNR}} on the calibration groups and apply it to the held-out groups. Hallucination is the positive class. Tables R1 and R2 report the confusion counts, thresholds, and operating metrics.

Table R1. Token-FAR confusion counts on held-out answers.

ModelDatasetTNFPFNTP
QwenTruthfulQA1,9901,228283499
QwenHaluEval-QA2235083444
QwenFELM-Science1316823
QwenFaithDial615184239560
LlamaTruthfulQA2,157940301602
LlamaHaluEval-QA2541951476
LlamaFELM-Science2271219
LlamaFaithDial578221232567

Table R2. Token-FAR operating thresholds and metrics.

ModelDatasetτ\tauCal.TestAP (%)Hall. (%)Prec. (%)Rec. (%)Spec. (%)Acc. (%)F1 (%)
QwenTruthfulQA0.73871,0004,00034.519.628.963.861.862.239.8
QwenHaluEval-QA0.704120080090.765.989.984.381.783.487.0
QwenFELM-Science0.6786156062.151.759.074.244.860.065.7
QwenFaithDial0.62864021,59880.650.075.370.177.073.572.6
LlamaTruthfulQA0.78311,0004,00039.422.639.066.769.669.049.2
LlamaHaluEval-QA0.706720080098.865.996.290.393.091.393.2
LlamaFELM-Science0.7403156071.551.773.161.375.968.366.7
LlamaFaithDial0.69904021,59879.250.072.071.072.371.771.5

Table R3. Held-out F1 (%) by method.

MethodQwen TruthfulQAQwen HaluEval-QAQwen FELM-ScienceQwen FaithDialLlama TruthfulQALlama HaluEval-QALlama FELM-ScienceLlama FaithDial
Perplexity23.784.550.065.138.591.248.159.4
LN-Entropy21.585.836.761.436.190.522.262.9
Energy35.670.031.856.846.169.660.656.0
P(True)35.184.565.864.940.776.548.163.9
Lexical Similarity32.68.244.457.147.162.050.848.1
Semantic Entropy27.870.665.962.735.771.761.061.5
EigenScore (aug.)36.54.052.554.938.925.665.155.8
NCI26.374.336.759.046.774.655.255.2
fDBD27.561.233.362.241.575.261.562.4
Token-FAR39.887.065.772.649.293.266.771.5
AuthorsOpenReview ↗

Response to Weakness 1, Questions 4 and 5, and Suggestion 1

W1, Q4, Q5, S1. Clarify the source and reliability of the labels on each benchmark and reconcile the FELM-Science counts with the public dataset.

Thank you for your constructive suggestions!

Candidate and label construction

  • TruthfulQA. We sample ten answers for each of 500 evaluation questions per model, giving 5,000 candidates. The intended automatic labeling rule marks an answer factual if its maximum ROUGE-L against the factual references exceeds 0.7 or its maximum MiniLM cosine similarity exceeds 0.5. We use the frozen automatic labels in the reported comparison.
  • HaluEval-QA. The qa_samples split provides the candidates and their binary labels. We evaluate one provided answer for each of 1,000 prompts.
  • FaithDial. Each of 1,000 evaluation turns contributes a benchmark-annotated hallucinated original and its faithful revision, yielding 2,000 candidates under the same histories and supplied knowledge.
  • FELM-Science. We use the 125 annotated response records in the Science domain. We reserve 50 records with entirely factual segment annotations for the dictionary and evaluate on the remaining 75 records, of which 37 are factual and 38 contain at least one false segment. The 50-reference budget leaves 75 records for evaluation, and 300 references would exceed the size of this domain. Table R4. Reference and evaluation splits.
DatasetReference recordsEvaluation recordsEvaluation answersAnswer source
TruthfulQA3005005,000Model-generated responses
HaluEval-QA3001,0001,000Benchmark-provided candidates
FELM-Science507575Annotated responses
FaithDial3001,0002,000Paired benchmark answers

Independent TruthfulQA label check

We added an LLM-as-a-judge audit on 120 responses, sampling 30 per model and original label. Gemini 4 Argon saw the question, response, and factual references while blinded to the original label. The judge gave a binary verdict for all 120 responses and agreed with the original labels on 103 (85.8% agreement, Cohen’s κ=0.717\kappa=0.717). Table R5 shows the label counts, including 17 disagreements.

Table R5. TruthfulQA agreement with Gemini 4 Argon.

Original automatic labelJudge factualJudge hallucinated
Factual555
Hallucinated1248

We also checked the automatic rule against the similarity scores stored for these 120 candidates. It reproduces 60 of 60 frozen labels for Llama and 60 of 60 for Qwen. The agreement figures above use the original frozen labels.

AuthorsOpenReview ↗

Response to Weakness 2, Questions 2 and 3, and Suggestion 5

W2, Q2, Q3, S5. Explain the reference supervision, teacher forcing, and what the experiments establish about the contribution and transferability of the dictionary.

Reference supervision and teacher forcing

We build one dictionary per model and dataset from known factual answers. A fixed-seed shuffle assigns 300 TruthfulQA questions to the reference set and the next 500 to evaluation. Evaluation answers enter the scorer as queries.

Teacher forcing feeds the reference prompt and known answer to the frozen LLM. We store each answer token’s state at its own position (Eq. 1). The content filter in §3.4 applies to candidate-score pooling. Binary decisions use labeled calibration data.

Reference quality, reconstruction, and transfer

  • Reference quality and affine centering. We paired factual and hallucinated answers to the same reference prompts and matched their token budgets. We also compared centered and uncentered ridge with the factual dictionary held fixed (Table R6).

Table R6. Effects of factual references and affine centering. Values are AUROC percentage-point differences with paired 95% intervals.

DatasetModelReference tokensFactual − hallucinated referencesCentered − uncentered ridge
HaluEval-QAQwen1,092+46.6 [43.5, 49.9]+1.4 [1.2, 1.7]
HaluEval-QALlama999+48.8 [45.3, 52.4]+0.2 [0.1, 0.4]
FaithDialQwen5,914+23.8 [22.3, 25.4]+0.2 [0.1, 0.2]
FaithDialLlama5,764+26.3 [24.7, 28.0]+0.2 [0.2, 0.3]
TruthfulQAQwen2,392+2.9 [0.0, 6.0]+0.9 [0.5, 1.2]
TruthfulQALlama2,374+5.0 [1.9, 7.7]+0.9 [0.6, 1.3]
  • Baselines using the same factual dictionary. We compared Token-FAR with four geometric scorers that use the same factual dictionary (Table R7). The strongest scoring rule depends on the model and dataset.

Table R7. Same-dictionary methods (AUROC %).

DatasetModelToken-FARLOFDiagonal MahalanobisGlobal PCA residualCosine 10-NN
TruthfulQAQwen62.159.855.152.659.3
TruthfulQALlama70.668.258.355.763.4
HaluEval-QAQwen91.794.191.387.681.7
HaluEval-QALlama97.197.996.795.393.5
FELM-ScienceQwen62.757.744.760.560.1
FELM-ScienceLlama70.569.857.874.270.5
FaithDialQwen80.574.764.176.379.6
FaithDialLlama78.276.054.473.777.8
  • Cross-dataset dictionary transfer. We replaced the in-domain dictionary with one from another dataset, keeping the model, target candidates, and scoring rule fixed. We removed reference states associated with exact normalized prompt matches and tested the full and matched token budgets (Table R8). AUROC fell in 11 of 12 transfer directions per model at each budget, showing sensitivity to the choice of reference dataset. Topic, answer format, and factual coverage vary together across datasets.

Table R8. AUROC change under dictionary transfer. Cells give full / matched token budgets relative to in-domain.

ModelReference sourceTruthfulQAHaluEval-QAFELM-ScienceFaithDial
QwenTruthfulQA—-50.7 / -48.8-7.3 / -10.0-21.8 / -20.7
QwenHaluEval-QA-12.8 / -13.7—-20.2 / -20.8-23.8 / -22.0
QwenFELM-Science+2.6 / +1.5-59.1 / -61.8—-16.4 / -15.8
QwenFaithDial-3.8 / -5.9-50.9 / -47.5-8.8 / -11.5—
LlamaTruthfulQA—-64.6 / -62.7-6.5 / -10.5-28.3 / -27.5
LlamaHaluEval-QA-27.2 / -26.3—-25.2 / -26.3-30.6 / -30.1
LlamaFELM-Science+6.9 / +8.7-71.0 / -69.0—-21.8 / -22.6
LlamaFaithDial-5.5 / -6.3-57.9 / -56.2-7.3 / -6.2—

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

W1 and Q1. How much of the performance comes from clean in-domain references versus affine reconstruction? How well does the dictionary transfer? Can a low residual arise from extrapolation beyond the retrieved neighbors?

Thank you for the careful review and for highlighting the residual decomposition. We added controls for reference quality, geometric scoring, and dictionary transfer. Qwen and Llama denote Qwen2.5-7B-Instruct and Llama-3.2-3B-Instruct below.

  • Reference quality and affine centering. We replaced factual answers with incorrect answers to the same reference prompts, matched the reference-token budgets, and held the evaluation candidates and scoring rule fixed. We also compared centered and uncentered ridge with the factual dictionary fixed. Factual references improved AUROC by 23.8–48.8 points on HaluEval-QA and FaithDial and by 2.9–5.0 points on TruthfulQA. Centering added 0.2–1.4 points across these six settings.

  • Geometric scorers with the same factual references. We compared Token-FAR with local outlier factor (LOF), diagonal Mahalanobis distance, global PCA residual, and cosine 10-NN. Token-FAR leads in five of eight settings. LOF leads on HaluEval-QA for both models, and PCA leads on FELM-Science for Llama.

Table R1. Answer-level AUROC (%) for the same-factual-dictionary comparison.

DatasetModelToken-FARLOFDiag. MahalanobisGlobal PCACosine 10-NN
TruthfulQAQwen62.159.855.152.659.3
TruthfulQALlama70.668.258.355.763.4
HaluEval-QAQwen91.794.191.387.681.7
HaluEval-QALlama97.197.996.795.393.5
FELM-ScienceQwen62.757.744.760.560.1
FELM-ScienceLlama70.569.857.874.270.5
FaithDialQwen80.574.764.176.379.6
FaithDialLlama78.276.054.473.777.8
  • Cross-dataset and cross-model transfer. We changed the source dataset while holding the model, target candidates, and scoring rule fixed. We removed exact normalized prompt matches and tested both full and matched reference-token budgets. AUROC fell in 11 of 12 directions for each model at each budget. For example, TruthfulQA references reduced AUROC on HaluEval-QA by 50.7 points for Qwen and 64.6 for Llama at the full budget. FELM-Science to TruthfulQA was the exception. In a separate HaluEval-QA diagnostic, learned cross-model alignment and shuffled anchors gave similar AUROCs in both directions. The evaluations support a dependence on the reference source.

  • Affine extrapolation. A low affine reconstruction residual can occur outside the convex hull of retrieved factual reference states, since affine coefficients sum to one but may be negative. To assess whether this extrapolation materially affects hallucination detection, we compare Token-FAR with a convex-hull-constrained reconstruction using the same retrieved neighborhoods, requiring nonnegative coefficients that sum to one. Across eight model–dataset settings, 53.1–68.7% of scored tokens have at least one negative affine coefficient. Nevertheless, replacing the affine score with the convex-hull residual changes answer-level AUROC by less than 0.1 percentage points in every setting. These results suggest that allowing affine extrapolation has limited influence on aggregate hallucination discrimination. However, a low reconstruction residual indicates geometric compatibility with the selected factual references rather than guaranteed factual correctness, and individual hallucinated tokens may still receive low residuals.

The Token-FAR implementation is available in the anonymous code repository.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2

W2 and Q2. Does the residual respond to factual errors when topic, wording, length, and coverage are controlled? How does averaging affect detection and localization when a long answer contains a short error?

  • Same-question, equal-length factual edits. We constructed 34 HaluEval-QA pairs that share a question and source context and differ in one factual span while keeping word count fixed. Token-FAR ranked the false answer higher in 33 of 34 pairs for Qwen and 34 of 34 for Llama. The paired design shows that the score can respond to a localized factual change under these controls.

Table R2. Example of a single factual edit in HaluEval-QA.

VersionAnswer excerptQwen scoreLlama score
Factual… win 30 games in one season.77.083.2
Edited… win 25 games in one season.78.184.5
  • Reference coverage with fixed answers. We built an additional HaluEval-QA test with 80 year questions in 40 reciprocal pairs. We kept the candidate text, prompt, background dictionary, and reference-token count fixed and switched which year's representations were added to the dictionary. When those additional references covered the incorrect year, the rate at which Token-FAR ranked the false candidate higher fell by 25.0 points for Qwen and 2.5 for Llama. This intervention shows that coverage can affect the ranking even with identical evaluation text.

  • Sparse errors in longer passages. Eq. (10) averages content-token residuals. If only mm of nn token residuals increase by δ\delta, the answer score increases by mδ/nm\delta/n. We tested 24 equal-word-count HaluEval-QA passage pairs of 30–109 words with one edited factual span. Mean pooling ranked the edited passage higher in 22 of 24 pairs for Qwen and 24 of 24 for Llama. We then ranked tokens within each edited passage using the same reference dictionary and neighbors. Token-FAR led on both Top-1 and localization AP for both models (Table R2). Its highest-scoring token fell inside the edited span in 4 of 24 passages for Qwen and 11 of 24 for Llama. Its AP exceeded LOF by 8.8 and 12.9 percentage points, respectively, and the local orthogonal residual by 10.0 and 30.0 points. The gains over the orthogonal score suggest that the full residual helps localize the controlled edits.

Table R3. Within-passage error localization on the 24 edited HaluEval-QA passages. Each cell reports Top-1 / average precision (%), averaged across passages.

ModelToken-FARLOFLocal orthogonal residual
Qwen16.7 / 19.38.3 / 10.54.2 / 9.3
Llama45.8 / 51.733.3 / 38.812.5 / 21.7
AuthorsOpenReview ↗

Response to Weakness 3

W3. Reconcile the FELM-Science counts with the public dataset, audit the TruthfulQA similarity labels, and clarify what Table 1 measures.

  • FELM-Science. The original count of 1,000 evaluation questions was incorrect. FELM has 847 annotated response records across all domains, including 125 in Science. We reserve 50 wholly factual records for the dictionary and evaluate on the remaining 75 records. Of those, 37 have only factual segment labels and 38 contain at least one false segment. The split is disjoint by response record.

  • TruthfulQA label construction. We use 300 reference questions and 500 separate evaluation questions. For each model, ten sampled answers per evaluation question yield 5,000 fixed candidates. The intended rule assigns a factual label if the maximum ROUGE-L against the provided factual answers exceeds 0.7 or the maximum MiniLM cosine similarity exceeds 0.5. Table 1 reports answer-level AUROC, using one score and one label per candidate. For TruthfulQA, the labels are the frozen similarity-based proxy labels.

  • Blinded label audit. We sampled 120 responses from distinct questions, stratified by evaluated model and original label (30 responses per stratum). Gemini 4 Argon assessed each question, candidate response, and its factual reference answers without access to the original automatic label. The judge agreed with the original labels on 103 of 120 responses (85.8%), yielding Cohen's κ=0.717\kappa=0.717, with 17 disagreements (Table R3). This indicates substantial agreement between the two labeling procedures, although it does not establish the factual correctness of every label. Furthermore, replaying the automatic labeling rule using the stored similarity scores reproduced all 60 frozen labels for Qwen and all 60 for Llama, confirming consistency with the specified labeling criteria on the audited samples.

Table R4. TruthfulQA agreement with Gemini 4 Argon.

Original automatic labelJudge factualJudge hallucinated
Factual555
Hallucinated1248

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.