acceptodds
Under review as a conference paper at ICLR 2027

Token-Level Evidential Support Is Linearly Decodable from the Residual Stream

Abstract

We replace hallucination detection with a relaxation that is decidable by construction: for each token a model generates, was the evidence for it present in the context, and does the model's internal state reflect that? Each item is a two-part question about a 10-K filing, with the supporting sentence for one part deleted and replaced by a length-matched sentence from another filing, alternating which part is affected. Because we delete it ourselves the label is exact rather than annotated, and because both parts are answered in one generation the two classes share prompt, document, question and output. A linear probe on mid-network activations ranks the supported part's tokens above the withheld part's at AUC, bootstrap interval over answers, against a bar of fixed before the data was collected. The probe also localizes, which a single score for a whole response cannot do. Thresholding its per-token score recovers the supported part's tokens at a token-level intersection over union of , against for next-token entropy read at the same position. At a matched scoring unit the probe leads entropy by of AUC per token and per part. Three pre-declared controls come out at chance, as does a randomly initialized placebo, and two positive controls come out high, which is what makes the battery falsifiable. One leak is real: a bag-of-words classifier on the answer text alone separates the conditions across answers at , so the answer-level gate is benchmarked against that rather than against chance. Within one answer the contrast is not recoverable from which tokens were generated, and that is the control the main result rests on. The probe reads support and not accuracy: answers whose evidence was present but whose value was wrong are indistinguishable from correctly answered supported ones, while entropy places those same answers at its own midpoint. This matters most on long documents, where the model stops flagging the problem itself. Padding the document from to tokens with the question and the removed evidence held fixed, the rate at which the model declines to answer falls from to . The depth at which the signal appears does not move with length, on a grid of four probed layers, refuting a hypothesis we had registered. What the probe provides is a ranking within an answer, not a calibrated absolute score, and a pre-registered test of locality failed. Every result here comes from a single B-parameter model, with the headline replicated on one other.

Primary area:
interpretability and explainable AI
Keywords:
Probes, Mechanistic Interpretability, Separability, AI

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_czCqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether evidential support for generated tokens can be decoded from LLM internal representations. The authors construct controlled two-part questions from financial filings, selectively removing supporting evidence to create labels without human annotation. A linear probe trained on residual-stream activations achieves 0.862 within-answer AUC across 493 evaluable answers, outperforming next-token entropy and the Lookback Lens baseline. Additional experiments examine token-level localization, distinguish evidential support from factual correctness, and investigate effects of context length and network depth.

Critical strengths
  1. The paper investigates an important research question: whether information about evidence availability is encoded in LLMs' internal representations. This is an interesting and important direction that could motivate further research into understanding how LLMs use contextual information. However I do have some issues with the analysis, see W1.

  2. The paper is well written, clearly organized, and easy to follow. The authors explain their methodology and experimental findings clearly, making the paper accessible and easy to understand.

Critical weaknesses
  1. To my understanding, the probe learns to assign a low support score to text for which evidence was not available in the context. Nowadays, LLMs generate much of their text from parametric knowledge, which means the probe could flag a large number of tokens. However, this is not particularly useful, since the real challenge then becomes distinguishing between factually incorrect tokens and correct tokens generated from parametric knowledge, which may still be valuable to the user. Maybe I am missing something, but I find the practical value of the proposed method unclear. If the authors could also provide a way to make this distinction (on top of flagging ungrounded tokens), their approach would be considerably more valuable.

  2. Even if I am missing something regarding the practical utility of the method, the empirical scope of the paper remains quite narrow, relying on a highly artificial experimental setup. The experiments are conducted on a single dataset and only two LLMs, with most evaluations performed on one model. This is considerably less comprehensive than what I would expect from an empirical paper at ICLR. I suggest extending the evaluation to additional datasets and domains, potentially using the same data construction methodology, as well as a broader range of LLMs to assess the generalizability of the findings.

  3. To be more convinced that the probe actually learns to distinguish grounded from ungrounded tokens, I would like to see some evidence of generalization beyond the specific two-part question construction. For example, testing the same probe on standard, single-question answers from financial reports (with proper annotations for grounded and ungrounded tokens), without retraining, would strengthen the authors' claim about really learning to distinguish grounded from ungrounded tokens

Questions for authors
  1. Connecting to W1: Given that the probe detects evidence availability rather than factual correctness, what practical applications do the authors envision, particularly when LLMs generate correct information from parametric knowledge?

  2. Connecting to W3: Would the probe generalize to a different question format, such as single-question answers from the same financial reports, without retraining? This would help establish whether it learns evidential support rather than patterns specific to the two-part construction.

Reviewer_wea2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a dataset of questions whose answers are either supported or unsupported by the given context. This dataset enables training a linear probe that distinguishes answer tokens whose justification is present in the context from answer tokens whose supporting evidence has been removed. The results show that the linear probe achieves higher within-answer AUC than a next-token entropy baseline.

Critical strengths
  • The experimental setup is clear and results are analyzed from several perspectives.

  • The dataset design is a good idea. Similar in spirit to the contrastive datasets used in activation-engineering work, it enables the identification of a feature in model activations without requiring annotation of LLM outputs. The authors provide a proof of concept that this dataset can be used to decode contextual evidential support from model activations.

Critical weaknesses
  • The Contributions of the paper. The title and introduction suggest that the main contribution is evidence that "contextual evidential support" is linearly decodable from LLM representations. However, the distinction between this finding and the results of prior work cited by the authors is not sufficiently clear. The related-work section emphasizes the token-level formulation, but the paper does not establish what new representation-level insight this granularity provides beyond existing probing approaches. In particular, it is unclear whether token-level probing identifies a substantially different signal from probes operating on aggregated activations, or instead measures a similar signal at a finer granularity. Token-level detection could offer advantages over a response-level score, but these advantages are not clearly motivated or demonstrated, and the paper lacks qualitative analysis illustrating their value. Consequently, the clearest contribution appears to be the contrastive dataset and its associated experimental results, rather than a distinct new finding about model representations or a demonstrated advance in hallucination detection.

  • Comparisons and practical utility: The paper does not claim that its dataset enables the extraction of better features or improved performance over existing methods. Nevertheless, the evaluation is limited to placebo baselines, ablations, and a badly introduced "Lookback Lens" whose results are not presented in a table or figure. These comparisons leave the method’s performance relative to established alternatives unclear. Related work may suggest that the approach could generalize beyond the proposed dataset, but the extent of this generalization remains an open question.

  • The fact that the deleted sentence always corresponds to the same semantic slot, whose gold values differ systematically from the retained slots, is a major confound rather than a minor limitation. It seems like alternating the deleted slot and matching slot properties would substantially strengthen the dataset.

Questions for authors
  • Did you conduct a qualitative analysis of the probe’s token-level scores on free-form question answering, beyond the structured numeric-answer setting?

  • Could you provide concrete examples of downstream use cases where token-level granularity would be practically beneficial, rather than text-level scoring?

  • Beyond its finer granularity, can you demonstrate that the token-level probe captures a qualitatively different signal from a global text-level probe? For example, through transfer, intervention, or analyses of what each probe encodes.

Reviewer_5nGXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses linear probing to investigate whether a model's hidden states encode whether evidence supporting the generated content is present in the context. The authors construct a dataset of two-question numerical QA tasks from 10-K annual reports by removing the supporting sentence for one question and replacing it with an irrelevant sentence of matched length. This provides exact ground-truth labels by construction, with both classes of tokens drawn from the same generation.

Using L2 logistic regression on the residual stream at layer 16 of Minitron-4B-Base, the probe achieves an answer-level AUC of 0.862 (exceeding the 0.70 threshold) and a token-level IoU of 0.545, compared with 0.461 for next-token entropy. Under the same evaluation metrics, the probe outperforms entropy (+0.164 AUC / +0.088 IoU) and Lookback Lens (+0.077 AUC).

The authors also conduct a set of preregistered negative and positive controls and argue that the probe detects evidence support rather than answer correctness. Key limitations include uncalibrated scores, a failed preregistered locality test, and evaluation on only one corpus. Results on Gemma-2-2B constitute a replication on the same corpus rather than validation across a different dataset.

Critical strengths
  1. Clean label construction: Exact labels are obtained by deleting and replacing supporting sentences, without requiring human annotation or a judge model. Both classes share the same prompt, document, question, and decoding process, controlling for confounding factors such as token position.

  2. Rigorous validation: The authors preregister evaluation thresholds, include multiple negative and positive controls, and transparently report failed preregistered tests and conclusions that were subsequently withdrawn.

  3. Distinction between evidence support and answer correctness: The study distinguishes whether an answer is supported by contextual evidence from whether it is factually correct. It also compares the probe with entropy and Lookback Lens using the same evaluation units, highlighting that these methods capture different types of information.

Critical weaknesses
  1. Limited novelty. Linear probes are already widely recognized as a means of extracting information about truthfulness, uncertainty, and contextual dependence from a model's internal representations. The method itself is standard L2 logistic regression, so the main contribution lies in the dataset construction. However, the value of this construction depends on whether it generalizes to real-world hallucination detection, which the paper does not demonstrate.

  2. Insufficient model coverage. The main results are obtained exclusively from Minitron-4B-Base. The second model, Gemma-2-2B, only replicates the headline metrics; the control experiments, baseline comparisons, and analyses of answer length and layer depth are not repeated. Both models are small base models in the 2B–4B range. Evaluation should be extended to larger models (e.g., 7B+) and instruction-tuned models, as the latter are more representative of practical hallucination detection scenarios.

  3. Lack of out-of-domain generalization evaluation. Both training and evaluation use the same corpus, template, and answer format (integers with at least three digits that appear verbatim in a single sentence). The authors themselves acknowledge the absence of cross-corpus transfer experiments. Table 2 of Lookback Lens shows that the AUROC of hidden-state probes can drop from above 90 to 56–59 when transferred across tasks, suggesting that high in-distribution AUC does not necessarily translate into practical utility.

Questions for authors
  1. Potential confounding from answer style. The probe is trained on sup0/sup2, where the entire answer belongs to a single class, but evaluated on sup1, where the answer contains a mixture of classes. Since the generation styles may differ between these settings, how can we verify that the probe has not learned answer-style cues rather than evidence support? Could a control experiment train on one subset of sup1 and evaluate on a disjoint subset of sup1?

  2. Potential reliance on shallow lexical signals. How can we rule out the possibility that the probe merely detects shallow cues, such as whether a numerical value can be found verbatim in the context? Could this be tested using paraphrased evidence, evidence distributed across multiple sentences, or cases where the evidence is present but the numerical value differs from the generated answer? For example, could the evaluation include negated statements or other linguistic constructions that require interpreting the semantic relationship between the evidence and the answer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.