acceptodds
Under review as a conference paper at ICLR 2027

Population-Conditioned Benchmark Item Influence Beyond Difficulty and Discrimination

Abstract

Benchmark item importance is often summarized by pass rate and discrimination. For model ranking, however, these summaries leave a population-conditioned component: which items matter depends on which models are being compared. We introduce population-conditioned ranking influence, a close-pair leverage statistic measuring which items distinguish similarly performing models within a specified population. Across nine benchmark matrices, residual fingerprints retain reproducible population-specific structure after controlling for population-local pass rate, pass rate squared, and corrected item–total discrimination, including under target-excluded capability definitions. On an exactly aligned 441-model by 12,032-item MMLU-Pro matrix, fully disjoint panels independently recover this structure: increasing panel size from 75 to 100 raises median within-population Spearman similarity from 0.361 to 0.435, while cross-population similarity remains much smaller, changing from 0.072 to 0.081. Fixed-margin Curveball randomization does not reproduce this agreement. The same disjoint-panel ordering appears across four additional large response matrices spanning scientific questions, multi-step reasoning, instruction following, and hard mathematics, with GPQA providing a second strong fixed-margin-null-confirmed result. Finally, signed item sets learned from one capability population preserve close-pair rankings of unseen models from that population better than sets learned from mismatched populations; all six paired contrasts are positive with 95% confidence intervals excluding zero. Together, these results show that benchmark ranking importance is relational rather than intrinsic: which questions matter depends on which models are being distinguished, and that population-specific structure predicts relative item-set utility on unseen models.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, language model evaluation, population-conditioned evaluation, ranking influence, benchmark item analysis, model comparison, close-pair ranking, evaluation reliability, item discrimination, benchmark robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nq7TOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper leverages existing response data to investigate which items can effectively differentiate models that have similar total scores. The authors discover that the impact of individual items exhibits a reproducible pattern after controlling for accuracy and item discrimination. Furthermore, selecting items tailored to a specific group proves more effective at preserving the relative rankings of held-out items

Critical strengths
  1. The experimental design is solid, considering the multiple control experiments.

  2. This research uses performance metrics outside the target benchmark, reducing the circular justification.

  3. The author clearly acknowledges the generalization scope and statistical control in the limitations section.

Critical weaknesses
  1. It is still unclear whether the observed effect can already be explained by a standard IRT model.

  2. The differences between the ability groups may partly come from differences in model families rather than ability itself.

  3. It is not clear whether the new metric is really necessary, since simpler baselines such as within-group random item selection are not included in the comparison.


No proper rebuttal during the discussion phase

Questions for authors
  1. Do simulated responses from a fitted IRT model show similar residual fingerprints?
  2. Do the results hold when the selection and test sets contain different model families?
  3. How much does the method improve over random selection within the same group, selection by discrimination, and selection using the metric without residualization?
Reviewer_dwWTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies "residual influence fingerprints": take the discriminability of an item within a capability population and regress out the population pass rate (and its square) as well as correlation between correctness and the population-local total score (the obvious determinants of influence when population is determined by capability) and rank by the residual. The paper shows that this ranking is stable (re-sampled/disjoint subsets of models yield largely preserved rankings) and distinct across these capability populations, across a range of benchmarks. They show that signed rankings (both matter and agree with overall benchmark) give a small number of items that much better predict heldout model performance from the same capability class vs a different capability class.

Critical strengths

This is a really interesting science-of-measurement paper that studies the stability of a residual ranking -- the stability is perhaps not completely surprising but the definition for the residual does a good job of screening out very obvious factors. I hence think this is well-motivated and should be interesting to a broad audience, and this positively affects my overall assessment.

Experimentation is extensive and well-motivated. For example, the fully-disjoint panel experiments are really helpful in establishing how stable this fingerprint is within a class. Broadly, the authors seem to have done a good job of anticipating a range of natural lines of inquiry for this residual object, and this strengthens my overall assessment.

Critical weaknesses

A main weakness is that this is only investigated for coarse model performance groupings, rather than other relevant populations (which, presumably, would require different regressors to form residuals over). This is mentioned in limitations, but it would certainly better illustrate the general phenomenon if experiments branched beyond these capability tiers as populations. While maybe not easy to address (and hence, factors into my score), I think this might be a matter of "that's for future work" given the amount of experimentation that was done.

Questions for authors

How does model family (especially e.g. some branching of the same base model) influence transfer to new models?

Reviewer_3L7ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that which benchmark items matter for ranking models depends on which models are being compared. Within a model population P, it collects close pairs of models whose total scores differ by a threshold, and defines an item's influence as the fraction of these close pairs that the item separates, which I think is quite novel. Validation covers breadth, recovery, and transfer parts.

Critical strengths

Measuring an item's importance by how often it separates close pairs of models within a specified population, rather than by global pass rate or discrimination, is a novel and natural way to ask which items decide close comparisons.

Critical weaknesses

My concern is the lack of impact for the community. The main takeaway is that ranking influence depends on which models are compared, so influence estimates should report their reference population. However, the idea that item usefulness depends on where models sit on the ability scale is already the basis of IRT item information and adaptive testing, and the premise of Fluid Benchmarking. The paper does not show a concrete case where population-conditioned influence changes a decision. The only downstream test (S7) gives close-pair accuracies close to chance, so practical value is not yet demonstrated.

Questions for authors

How are model families (base/instruct/size variants of the same base model) distributed across panels and splits? Do the results hold with family-disjoint panels and splits, and with model- or family-level resampling for the intervals? Can you show examples of items with large residuals in each population? Maybe in the future submission, I would suggest separating shared and population-specific components, and extending the analysis to graded and open-ended evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.