- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper raises a question about LLM evaluation: models may achieve different benchmark performance because of differences in their training data, and familiarity with benchmark inputs may inflate accuracy. It therefore proposes a two-dimensional diagnostic for benchmark evaluation that considers both accuracy and the distance between test inputs and training data. Specifically, the paper defines distributional distance using embedding distances to neighboring training examples. Since training corpora are usually unavailable, it uses perplexity and Min-K% as proxies for this distance. The paper empirically evaluates these proxies and assesses nine models across 13 benchmarks, reporting generally negative correlations between benchmark accuracy and estimated distributional distance.
Critical strengths
- I think the motivation of the paper makes sense.
- The presentation is clear and it conducts extensive experiments.
Critical weaknesses
- The definition of byte-normalized perplexity seems problematic. is defined as . If is the negative log-probability of the entire token, should this instead be (line 166)?
- Fundamentally, I am not convinced that the current distance metrics separate familiarity from capability. For example, if two models are trained on the same corpus, but model A learns better than model B, model A can achieve both lower perplexity and higher benchmark accuracy. This gain comes from learning better rather than differences in training-data exposure.
- The validation experiment comparing the distance proxies with k-NN distance is also problematic. The paper’s cross-model interpretation requires evidence that “if model A finds a benchmark more surprising than model B does, this indicates that the benchmark is farther from A’s training data than from B’s.” However, the current evaluation only demonstrates that “within a given trained model, texts with greater nearest-neighbor distance often also have higher input surprise.” For example, both models might rank easy and difficult texts similarly, while model A assigns lower probabilities to the evaluated inputs because it is less capable. Their within-model correlations could both be strong, even though comparing the absolute proxy values across models would be misleading.
- Simply considering average accuracy and average distance is also problematic. A model could have a higher overall distance while achieving its accuracy advantage primarily on low-distance samples. Therefore, higher average accuracy together with higher average distance does not necessarily demonstrate better performance on distant samples, and the proposed criterion does not resolve this issue.
Questions for authors
See weaknesses.