acceptodds
Under review as a conference paper at ICLR 2027

Prediction Training, Representation Geometry, and Hallucination Detection

Abstract

Prediction training shapes the relative geometry of hidden representations. For an attention model trained by gradient descent, we characterize how representations of contexts associated with the same prediction approach one another. Their remaining differences follow a limiting direction determined by prediction costs, context composition, and the parameter update. This connects prediction training to pairwise representation geometry. We use this component-mixture perspective to develop a pre-generation hallucination detector. For each labeled outcome, the detector reconstructs a test state by reweighting nearby reference states. Its energy penalizes concentrated mixture weights and reconstruction error. It requires no classifier training. The detector achieves a mean AUROC of 79.26% across 21 language and vision-language model–dataset combinations. On visual tasks, it reaches 81.65%, exceeding a supervised pre-generation probe by 2.35 points on the same independent test sets.

Primary area:
interpretability and explainable AI
Keywords:
Representation Geometry, Uncertainty Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8V6kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper has two parts. The first part is theory. It studies a simplified attention model. It shows that when two inputs need to predict the same target, training makes their internal representations closer. The gap shrinks at a rate of O(1/k), where k is the number of training steps. A small difference remains, and its direction can be predicted. Synthetic experiments confirm this. The second part is a hallucination detector. It works before the model generates an answer. The detector keeps a set of reference questions, each labeled by whether the model answered it correctly. For a new question, it checks whether the question's hidden state looks more like the correct references or the incorrect ones. This gives a risk score. Experiments use three LLMs and three VLMs, with 21 model–dataset pairs in total. The average AUROC is 79.26.

Critical strengths
  1. The theory is solid. The authors state their assumptions clearly and give complete proofs. Their synthetic experiments match the theory well.

  2. The detector is simple and cheap. It only needs the model to read the question once. It does not need to generate multiple answers. It works on both text-only and vision–language models, and it beats kNN voting in all 21 settings.

Critical weaknesses
  1. The baselines are too weak. The detector uses 1,000 labeled examples and tuned hyperparameters, so it is basically a classifier. It should be compared with other simple classifiers trained on the same data, such as tuned kNN or logistic regression. Instead, the paper compares with an untuned kNN and probe. This is my main concern.

  2. The results are not tested for robustness. The gains over the probe are small, and all results come from a single random split without confidence intervals. There is also no cross-dataset test, so the detector may only learn which types of questions the model often gets wrong.

  3. The theory and the detector are not really connected. The theory only explains how training makes representations similar. It does not say anything about when the model gives a wrong answer. The authors also say that the detector works without the theory's assumptions. So the theory does not explain why the detector works.

Questions for authors
  1. The current experiments are not systematic enough to support the detector's advantage. Could you provide a more complete evaluation? This should include tuned simple baselines that use the same labeled data, results over multiple random splits with confidence intervals, and tests across datasets. It would also help to show how the results change with the size of the reference set.

Overall, I think the current evidence is not sufficient to support the paper's main claims.

Reviewer_fZVpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows that under prediction training, representations with the same target converge to each other at rate 1/k, and their remaining difference follows a predictable direction. Building on this, it proposes a training-free pre-generation hallucination detector. The detector compares how well correct-answer reference states and incorrect-answer reference states each reconstruct a test state.

Critical strengths
  • Pre-generation detection is practically useful.
  • The method is simple and training-free.
  • The evaluation is broad, covering LLMs and VLMs on seven datasets.
Critical weaknesses
  • The theory does not justify the method. The detector does not use Theorem 1. The authors themselves state that the detector does not require the assumptions of Section 3.
  • The kNN comparison is unfair. kNN uses only 5 neighbors, while the proposed method uses 50–200 neighbors per class with tuned hyperparameters. With 5 neighbors, the score takes only six possible values, which structurally disadvantages kNN under AUROC.
  • Section 4 is hard to follow. The symbols in Figure 2 (i, a, and the coordinates) are not defined near the figure.
  • The presentation is incomplete. There is no conclusion or limitations section; Section 7 is one short paragraph that restates the results. Key definitions (the model in Appendix A and the Figure 2 setup in Appendix G) appear only in the appendices, so the main text is hard to follow on its own. Limitations are also not discussed anywhere: the paper does not address the strong assumptions of the theory, the need for about 1,000 labeled references per task, the cost of solving an optimization problem per sample, or the single-token-answer case (see the second question).
  • Limited novelty. Prior work already estimates a model’s knowledge or error risk from pre-generation states [1] and shows that same-target representations converge [2]. The paper neither cites nor discusses them.

[1] Gottesman & Geva. Estimating Knowledge in Large Language Models Without Generating a Single Token. EMNLP, 2024. [2] Zhao et al. Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations. arXiv:2408.15417, 2024.

Questions for authors
  • Does the gain remain when compared with distance-weighted kNN using the same number of neighbors (50–200), or with the distance to class-wise local means? Only this comparison can support the claim that “reconstruction uses information beyond the labels.”
  • For tasks with single-token answers (MMLU, TruthfulQA, CV-Bench, POPE), how does the method compare with a score based only on the probability of the first answer token? This state lies right before the output head, so it nearly contains the answer probability itself.
  • The reference and test sets are random splits of the same dataset. Does detection still work with references from a different dataset? In practice, labeled data are hard to obtain for every task.
Reviewer_4nkSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that when two contexts share the same prediction, the representations of those contexts approach one another. They leverage this observation to develop a hallucination detector. The detector score achieves a mean AUROC of 79.26% across 21 language and vision-language model and dataset combinations, indicating that it is effective.

The authors introduce a local mixture energy metric as a score for hallucination detection. They show that if attention model values and a prediction head are both fixed, then two contexts with the same prediction target will converge in representation. Synthetic experiments in which keys, values, and the prediction head of an attention layer are fixed confirm the theoretical argument. The synthetic expeirments with transformer models also support the theory for the common case the values, keys, prediction heads etc are not frozen. The energy metric measures how well hidden state representations belonging to either true or false statements reconstruct the test hidden state. There is also a penalty on departure from the uniform mixture.

The method performs well compared to simple baselines such as knn classifiers, activation probes, lexical similarity, eigenscore, etc on a variety of language and vision-language tasks.

Critical strengths

The paper targets an important problem and is well-motivated. It is grounded in literature and has sufficient treatment of related work.

It provides a genuinely interesting, novel geometric approach to hallucination detection that is supported by theory and synthetic experiments.

The method shows strong performance on a variety of datasets compared with several baselines.

Overall, I think this paper presents an interesting, novel idea, and if the weaknesses are addressed, I'd be inclined to improve my score.

Critical weaknesses

The paper lacks a comparison of the computational cost of this method with that of similar methods. This may be important in scenarios where hallucinations must be detected on the fly.

It is not clear if this method could be applied effectively to long-form generations (like chain-of-thought). Clarifying this could make the method more attractive as a practically useful approach.

The representation power of the probe was not clear from the paper. Is this a linear/logistic regression model? Would a more powerful, nonlinear model achieve better scores?

Questions for authors

Figure 3 is good evidence to support the paper’s claims. Does the result of the synthetic experiment in figure 3 hold with deeper transformer models? Does the signal dissipate or stay roughly the same?

The word “value” is overloaded and can be confusing. In 215, does “value” mean a value to of an answer to a question/prompt? I believe it does, but some clarification would help other readers.

The explanation following “shared components after full-parameter training.” is confusing – please consider clarifying.

How well does this method work for more complex, longer answers? If a hallucination occurred deep in a response, it is not clear that this method could capture it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.