- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper has two parts. The first part is theory. It studies a simplified attention model. It shows that when two inputs need to predict the same target, training makes their internal representations closer. The gap shrinks at a rate of O(1/k), where k is the number of training steps. A small difference remains, and its direction can be predicted. Synthetic experiments confirm this. The second part is a hallucination detector. It works before the model generates an answer. The detector keeps a set of reference questions, each labeled by whether the model answered it correctly. For a new question, it checks whether the question's hidden state looks more like the correct references or the incorrect ones. This gives a risk score. Experiments use three LLMs and three VLMs, with 21 model–dataset pairs in total. The average AUROC is 79.26.
Critical strengths
-
The theory is solid. The authors state their assumptions clearly and give complete proofs. Their synthetic experiments match the theory well.
-
The detector is simple and cheap. It only needs the model to read the question once. It does not need to generate multiple answers. It works on both text-only and vision–language models, and it beats kNN voting in all 21 settings.
Critical weaknesses
-
The baselines are too weak. The detector uses 1,000 labeled examples and tuned hyperparameters, so it is basically a classifier. It should be compared with other simple classifiers trained on the same data, such as tuned kNN or logistic regression. Instead, the paper compares with an untuned kNN and probe. This is my main concern.
-
The results are not tested for robustness. The gains over the probe are small, and all results come from a single random split without confidence intervals. There is also no cross-dataset test, so the detector may only learn which types of questions the model often gets wrong.
-
The theory and the detector are not really connected. The theory only explains how training makes representations similar. It does not say anything about when the model gives a wrong answer. The authors also say that the detector works without the theory's assumptions. So the theory does not explain why the detector works.
Questions for authors
- The current experiments are not systematic enough to support the detector's advantage. Could you provide a more complete evaluation? This should include tuned simple baselines that use the same labeled data, results over multiple random splits with confidence intervals, and tests across datasets. It would also help to show how the results change with the size of the reference set.
Overall, I think the current evidence is not sufficient to support the paper's main claims.