acceptodds
Under review as a conference paper at ICLR 2027

A Distributional Distance Perspective on Evaluating LLMs

Abstract

Large language models (LLMs) are commonly evaluated by accuracy on public benchmarks. Since different LLMs are trained on different data distributions, raw accuracy may conflate model capability with training-data proximity, leading to unfair evaluations. The first factor is the well-studied data contamination. This paper explores another factor triggering unfair evaluation from the perspective of distributional distance between the input data and the LLM training data. We first adopt simple approximation methods to measure the distributional distance, and then empirically verify a strong statistical correlation between accuracy and distributional distance. We therefore introduce a two-dimensional criterion that jointly considers accuracy and distributional distance, enabling fairer evaluations. Finally, we re-evaluate nine LLMs on thirteen standard benchmarks and obtain several interesting findings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Evaluation, Distributional Distance, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hmvJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper raises a question about LLM evaluation: models may achieve different benchmark performance because of differences in their training data, and familiarity with benchmark inputs may inflate accuracy. It therefore proposes a two-dimensional diagnostic for benchmark evaluation that considers both accuracy and the distance between test inputs and training data. Specifically, the paper defines distributional distance using embedding distances to neighboring training examples. Since training corpora are usually unavailable, it uses perplexity and Min-K% as proxies for this distance. The paper empirically evaluates these proxies and assesses nine models across 13 benchmarks, reporting generally negative correlations between benchmark accuracy and estimated distributional distance.

Critical strengths
  1. I think the motivation of the paper makes sense.
  2. The presentation is clear and it conducts extensive experiments.
Critical weaknesses
  1. The definition of byte-normalized perplexity seems problematic. DistPPL\mathrm{Dist}{\mathrm{PPL}} is defined as exp⁡(∑tbtatB)\exp\left(\frac{\sum_t b_t a_t}{B}\right). If at=−log⁡p(xt∣x<t)a_t=-\log p(x_t\mid x{<t}) is the negative log-probability of the entire token, should this instead be exp⁡(∑tatB)\exp\left(\frac{\sum_t a_t}{B}\right) (line 166)?
  2. Fundamentally, I am not convinced that the current distance metrics separate familiarity from capability. For example, if two models are trained on the same corpus, but model A learns better than model B, model A can achieve both lower perplexity and higher benchmark accuracy. This gain comes from learning better rather than differences in training-data exposure.
  3. The validation experiment comparing the distance proxies with k-NN distance is also problematic. The paper’s cross-model interpretation requires evidence that “if model A finds a benchmark more surprising than model B does, this indicates that the benchmark is farther from A’s training data than from B’s.” However, the current evaluation only demonstrates that “within a given trained model, texts with greater nearest-neighbor distance often also have higher input surprise.” For example, both models might rank easy and difficult texts similarly, while model A assigns lower probabilities to the evaluated inputs because it is less capable. Their within-model correlations could both be strong, even though comparing the absolute proxy values across models would be misleading.
  4. Simply considering average accuracy and average distance is also problematic. A model could have a higher overall distance while achieving its accuracy advantage primarily on low-distance samples. Therefore, higher average accuracy together with higher average distance does not necessarily demonstrate better performance on distant samples, and the proposed criterion does not resolve this issue.
Questions for authors

See weaknesses.

Reviewer_X1r7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses perplexity and low-probability word scores to approximate the distance between the input and the training distribution, and advocates reporting these metrics together with accuracy to differentiate the model's generalization ability and familiarity with the input.

Critical strengths

It transforms the question of "familiarity with training data may affect evaluation results" into a testable problem and provides a diagnostic perspective beyond accuracy.

Critical weaknesses

There is a clear inconsistency between the formula definition and the table figures. Page 4 defines PPL as the exponent of the average negative logarithmic probability. According to this formula, it should be no less than 1, but the table on page 7 shows 0.87, 0.77, and 0.69. If these are logarithmic or otherwise transformed indicators, this needs clarification; otherwise, at least one part of the formula, label, or calculation needs correction.

The new ranking formula mechanically penalizes the model with the smallest distance. Page 25 first normalizes the distance to 0-1 before calculating "accuracy × distance". Therefore, the model with the smallest distance will have a total score of zero, no matter how high its accuracy is; thus, the occurrence of a large number of ranking reversals does not prove that the new ranking is closer to the true generalization ability.

Overall, I think this paper is valuable after the revision.

Questions for authors

See weaknesses.

Reviewer_X6sTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper study the bias in accuracy based LLM evaluation arising from distribution differences between testset and training set. The authors propose using PPL and Min-k% as distributional distances without requiring access to pretraining data, and validate their correlations with k-nearest-neighbor distances through controlled experiments.They evaluate nine LLMs and 13 benchmarks and observe strong negative correlations between accuracy and distance. So the author suggests that higher performance may partly reflect greater input familiarity rather than generalization. Based on these results, the paper proposes a two-dimensional evaluation framework that jointly considers accuracy and distance, revealing substantial differences from conventional accuracy-only model rankings.

Critical strengths
  1. The new evaluation metric GS proposed in this paper is interesting and useful for evaluating the model's true generalization ability. This metric can reverse the rank based on performance.

  2. The writing is good, and the implementation details are good for reproduction.

Critical weaknesses
  1. PPL and Min-k% are inherently influenced by model capability, not just training-data familiarity. Even when two models share identical training data, a stronger model may achieve lower PPL and higher accuracy, making it hard to tell whether the observed negative correlation reflects distributional proximity or general modeling ability. So I think the correlation between accuracy and PPL (Min-k%) also depends on model architecture, training hyperparameters, and tokenizers.

  2. A model may assign high likelihood to previously unseen inputs because it has learned generalizable patterns, rather than because similar examples appeared in its training corpus. So, the proposed metrics may conflate genuine generalization with training-data familiarity, potentially undermining the distinction the paper aims to establish.

  3. GS multiplies accuracy by min-max normalized distributional distance, which automatically assigns a score of zero to the model with the smallest distance, regardless of its accuracy. This formulation may systematically penalize stronger models and produce rank reversals driven by the normalization scheme rather than genuine differences in generalization ability.

Questions for authors

If we first train a model on all data similar to a set of downstream benchmarks, how much additional improvement can we get by further training it on high-quality but dissimilar data? This is just an open-ended question I'm curious about, so no worries if you don't have an answer!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.