- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when random subsets of an LLM benchmark preserve the full-benchmark model ranking. It combines population-relative item discrimination with the classical Spearman–Brown relationship to predict ranking agreement and estimate evaluation budgets.
Critical strengths
The proposed predictor is easy-to-use, interpretable, and requires no fitted coefficients.
Critical weaknesses
Related work already studies evaluation size, ranking reliability, and population dependence [1,2]. [2] is particularly relevant but is not discussed in Sec. 2. The paper’s main distinctive contribution is therefore using as a compact predictor. However, the experiments in Sec. 5 establish its predictive accuracy without comparing it against straightforward alternatives. Since the source models already have full evaluations, repeated random subsampling on their cached responses could directly estimate ranking agreement at each budget. A comparison under the same source-data budget would clarify whether provides additional accuracy, stability, or computational benefits. Moreover, Sec. 5.3 does not directly validate whether budgets selected through Eq. 8 achieve the requested agreement on target models. Thus, the reported correlations support the usefulness of the predictor, but its additional practical value remains unclear.
[1] SubLIME: Subset Selection via Rank Correlation Prediction for Data-Efficient LLM Evaluation. ACL '25.
[2] How Reliable is Language Model Micro-Benchmarking? ICLR '26.
Questions for authors
-
When Eq. 8 selects a budget for a target agreement of 0.9, what agreement is actually achieved on target models? It's better to report deviations from the target, particularly for low- instances and small source pools.
-
Sec. 4 explicitly distinguishes latent-ability reliability from agreement with the full-benchmark ranking. How should the ceiling and “unreachable targets” in Sec. 5.3 be interpreted for the objective in Eq. 3, given that evaluating all questions necessarily reproduces the full ranking?
6 comments
Response to W1.3 + Q1
W1.3 + Q1 Agreement achieved by budgets from Eq. 8
We ran this validation. We estimate on source models, compute the budget for a target agreement (Eq. 8), sample questions, and measure the agreement of disjoint target models with their full-benchmark ranking (the objective in Eq. 3):
| Target | Achieved agreement | Deviation |
|---|---|---|
| 0.90 | 0.899 | −0.001 |
| 0.95 | 0.952 | +0.002 |
- On feasible instances (), the budget from Eq. 8 gives the requested agreement on target models that were not used to estimate .
- For 15% of benchmark–population pairs, . Here the method reports that the target is above the predicted ceiling instead of returning a budget (see Q2).
- Low- instances. Where a budget exists, it is conservative on average: achieved agreement is 0.93 for a target of 0.90. For most low- instances, , so no budget is returned.
- Small source pools. The results hold for source pools of and models.
- Added as a budget-validation paragraph in §5.3 of the revised PDF.
Response to Q2
Q2 Interpreting the ceiling and "unreachable targets" under Eq. 3
We agree this needed clarification. Eq. 3 measures agreement with the finite full-benchmark ranking, which reaches 1 at . The revised PDF now separates two objectives:
- Reproducing the full-benchmark ranking (Eq. 3). There is no ceiling below 1. We add a finite-population form, , which equals 1 at and predicts observed full-benchmark agreement at (MAE ).
- Reproducibility on an independent question set. The ceiling predicts how reliably the full benchmark measures latent ability, which approximates how well its ranking would reproduce on new questions. It agrees with an independent split-half estimate at (MAE ). This supports it as a ceiling for reproducibility, not for Eq. 3.
- "Unreachable targets." A target above the ceiling is unreachable only for the reproducibility objective, using the available questions. It does not mean the full-benchmark ranking cannot be reproduced. The 15% of pairs with are those predicted unable to reach the target latent reliability with at most questions.
- Clarified in the §4, §5.3, and Appendix A (derivation of ) of the revised PDF.
Thank you + W1.1
Thank you for the careful review. We answer each point below and have updated the revised PDF accordingly.
W1.1 Related work and our contribution
We agree that our main contribution is using population-relative discrimination D as a compact predictor. We have expanded §2 of the revised PDF to discuss both works and to state this contribution directly: the predictor is parameter-free, closed-form, and depends only on D and the budget k.
[1] Saranathan et al. SubLIME: Subset selection via rank correlation prediction for data-efficient LLM evaluation. ACL 2025.
[2] Yauney, Warraich, and Swayamdipta. How reliable is language model micro-benchmarking? ICLR 2026.