Surrogate-Augmented Lee-Valiant Estimation for Adaptive Benchmark Evaluation

Evaluating modern models on large benchmarks is costly. We introduce the Adaptive Surrogate-Augmented Lee-Valiant estimator (ASA-LV), which combines historical-response prediction, adaptive querying, propensity-corrected residual contributions, and robust Lee-Valiant aggregation. We establish finite-sample error bounds under adaptive querying and reuse of queried answers for surrogate fitting. The bounds account for surrogate-class complexity and covering approximation error, yielding a $\tilde{O}(k^{-1/2})$ estimation-error rate under the stated assumptions. We evaluate ASA-LV on a collection of 19 benchmarks and in the FAQ evaluation setting comprising MMLU-Pro and a composite benchmark. At query budgets of 5% and 10%, ASA-LV achieves the lowest root-mean-square error (RMSE) and mean absolute error (MAE), together with the highest Kendall and Spearman model-ranking correlations, among the compared estimators in both settings. Relative to Factorized Active Querying (FAQ), the RMSE reductions are 10.517%/6.468% on the 19-benchmark collection and 14.927%/10.781% on the FAQ data, respectively. On the FAQ data, ASA-LV also achieves lower RMSE than FAQ, two augmented inverse-probability-weighted (AIPW) variants and random sampling when 5% or 10% of target answers are flipped before querying, under both scoring targets and both query budgets.