- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether a language model's hidden representations can help predict if it will answer a given input correctly. The authors propose a simple metric called typicality distance, which measures how different an input's hidden representations are from those of a reference dataset. The method does not require labeled data or additional model training.
Experiments on different language models and benchmarks show that inputs with smaller typicality distances are more likely to be answered correctly. The authors also compare their method with existing baselines and study how different factors, such as dataset size, model layers, input length, and statistical assumptions, affect the results.
Critical strengths
-
The paper applies information-theoretic typicality to hidden representations, providing a different way to study when language models are likely to produce correct answers.
-
The proposed approach requires no labeled calibration data, additional training, or auxiliary models. The consistent trends across different models and tasks suggest that representational typicality is a useful direction for further study.
Critical weaknesses
-
The paper divides examples into five buckets and measures the correlation between typicality distance and bucket-level accuracy. Even a perfect correlation across five buckets does not necessarily indicate reliable predictions for individual inputs. Since input-level prediction is the paper's central claim, metrics such as AUROC, AUPRC, and risk-coverage curves would provide stronger evidence.
-
Most experiments use calibration and test data from the same benchmark. It is unclear whether the method works across different datasets or distribution shifts.
-
The paper uses different activation normalization methods during calibration and testing, without analyzing their effects. More discussion of numerical stability, computational cost, and comparisons with stronger uncertainty estimation baselines would improve the evaluation.
Questions for authors
-
Can the authors report AUROC, AUPRC, and risk-coverage curves to show how well typicality predicts correctness for individual inputs, rather than grouped examples?
-
Does the method still work when calibration and test data come from different datasets? How does distribution shift affect the results?
-
Why are different standardization methods used during calibration and testing? How sensitive are the results to this choice and covariance regularization?