acceptodds
Withdrawn from ICLR 2027

Surrogate-Augmented Lee-Valiant Estimation for Adaptive Benchmark Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating modern models on large benchmarks is costly. We introduce the Adaptive Surrogate-Augmented Lee-Valiant estimator (ASA-LV), which combines historical-response prediction, adaptive querying, propensity-corrected residual contributions, and robust Lee-Valiant aggregation. We establish finite-sample error bounds under adaptive querying and reuse of queried answers for surrogate fitting. The bounds account for surrogate-class complexity and covering approximation error, yielding a estimation-error rate under the stated assumptions. We evaluate ASA-LV on a collection of 19 benchmarks and in the FAQ evaluation setting comprising MMLU-Pro and a composite benchmark. At query budgets of 5% and 10%, ASA-LV achieves the lowest root-mean-square error (RMSE) and mean absolute error (MAE), together with the highest Kendall and Spearman model-ranking correlations, among the compared estimators in both settings. Relative to Factorized Active Querying (FAQ), the RMSE reductions are 10.517%/6.468% on the 19-benchmark collection and 14.927%/10.781% on the FAQ data, respectively. On the FAQ data, ASA-LV also achieves lower RMSE than FAQ, two augmented inverse-probability-weighted (AIPW) variants and random sampling when 5% or 10% of target answers are flipped before querying, under both scoring targets and both query budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Evaluation, Active Testing, Finite-Sample Guarantees

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SvzLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ASA-LV estimates full-benchmark accuracy from few-answer queries. It reconstructs propensity-corrected contributions with a terminal surrogate and applies Lee–Valiant aggregation. The analysis allows adaptive querying and reuse of queried answers for surrogate fitting. Experiments cover 19 benchmarks and the FAQ data.

Critical strengths
  1. Estimating full-benchmark accuracy with few-answer queries addresses an important practical problem in model evaluation. The estimator itself is simple. It reconstructs augmented contributions using the terminal surrogate and then applies LV aggregation.

  2. The analysis allows adaptive querying and reuse of queried answers for surrogate fitting, and quantifies the effects of surrogate-class complexity and covering approximation. The evaluation covers 19 benchmarks and the FAQ data. ASA-LV reports lower RMSE and MAE than the compared estimators at both query budgets.

Critical weaknesses
  1. The main argument is difficult to follow. The introduction places propensity-amplified residuals alongside the data dependence introduced by terminal fitting, without first explaining why terminal reconstruction is needed. The motivation for combining these components, and the contribution relative to FAQ, remain unclear.

  2. The abstract gives an estimation-error rate without defining k. Section 3 then introduces many symbols and conditions through dense inline mathematics. The problem is simple, but its formulation places an unnecessary burden on the reader. Section 4 introduces dense technical details before making clear what the authors want to do and why. I found these explanations distracting because they obscured the main idea before I could grasp the intuition behind the method. Section 5 gives a proof outline and a short explanation of the nonlinear correction, but little guidance on the importance of the results. It does not provide a coherent interpretation of the error terms or clearly connect the two theorems to the experimental configuration. The derivations are difficult to assess as contributions.

Questions for authors
  1. Why reconstruct all contributions with the terminal surrogate rather than retain FAQ’s query-time predictors? What benefit justifies the resulting data dependence?

  2. Why use LV aggregation instead of simply averaging the augmented contributions? What benefit does it provide in this setting? Is it worth it? Is it so important that it should be the main contribution and appear directly in the paper's title?

  3. What do the authors want readers to learn from the theoretical analysis? According to my understanding, the authors intend to enhance the conclusion that LV makes sense and bias caused using a terminal surrogate is controlled. Why not organize the sections more clearly?

  4. Do the experiments separately establish the benefits of terminal reconstruction and LV aggregation over FAQ? How much does each change contribute to the overall improvement?

Reviewer_E47tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Adaptive Surrogate-Augmented Lee–Valiant estimation (ASA-LV) for estimating a model’s accuracy on a fixed benchmark using a limited number of adaptive queries. The method combines historical-response surrogate predictions with propensity-weighted residual corrections, reconstructs contributions using a final fitted surrogate, and applies Lee–Valiant aggregation. Its theoretical contribution addresses adaptive sampling and reuse of queried answers for surrogate fitting through uniform finite-sample error bounds over a predetermined surrogate class, together with a sharper variance-sensitive guarantee under additional assumptions. The method is evaluated on a collection of 19 benchmarks and the FAQ evaluation datasets at limited query budgets. Validation includes comparisons with existing estimators, alternative aggregation methods, and analyses of surrogate choices, query policies, and answer-record corruption.

Critical strengths
  1. The paper addresses an important statistical complication introduced by terminal refitting.
    FAQ constructs its contributions using the predictor available before each query. Recomputing earlier contributions with a predictor fitted after all queries changes the dependence structure, so the original conditional-mean argument cannot simply be reused. The proposed uniform analysis over a predetermined surrogate class addresses this issue explicitly, and the treatment of previously observed answers under sampling without replacement is clearly specified.

  2. The theoretical analysis involves more than a direct combination of existing components.
    The authors jointly handle adaptive dependence, a preliminary estimate formed from chronological blocks, and the global calibration used by LV, before extending the result to the terminal fitted surrogate. These features are not directly covered by the original iid analysis of LV. The explicit variance assumptions and the distinction between an oracle error guarantee and a computable confidence interval also help define the scope of the contribution.

  3. The improvements over the principal evaluation baseline are reasonably consistent.
    At the 5% query budget, ASA-LV achieves lower RMSE than FAQ on all 19 local benchmarks. Improvements remain after excluding the four benchmarks responsible for the largest gains. Comparisons between aggregators using the same contribution values, together with disclosure of tuning and model-identity limitations, provide useful supporting evidence.

Critical weaknesses
  1. The experiments do not fully isolate the benefits of terminal refitting from those of LV aggregation.
    Both contribution reconstruction and aggregation differ from FAQ. The overall improvement over FAQ is approximately 6.5–14.9%, while the advantage of LV over the strongest alternative robust aggregator on the same contributions is approximately 0.7–1.4%. These comparisons cannot be subtracted to determine component-level contributions. An explicit arithmetic mean of the terminally reconstructed contributions is needed to assess how much additional benefit comes specifically from LV.

  2. The connection between the theoretical guarantees and the experimental operating points remains incomplete.
    Theorem 1 controls the change introduced by LV relative to the contribution mean, but does not establish that LV has lower MSE or greater robustness than that mean. The sharper result uses confidence-dependent settings and a variance envelope, whereas the main experiments use fixed parameters. The paper does not demonstrate whether the resulting numerical bounds are informative at the evaluated sample sizes. This limits the practical interpretation of the theory without contradicting its stated oracle scope.

  3. The evidence for transfer to new model populations is limited.
    Model identities remain unverified for 12 local benchmarks, and some targets appear in both development and evaluation. The reported intervals primarily characterize query randomness for fixed histories and target cohorts, rather than variation across genuinely new model families. These limitations neither demonstrate leakage nor invalidate the conditional finite-bank guarantees, but they weaken broader claims about deployment generalization.

Questions for authors
  1. Could you provide a controlled comparison of the four relevant estimator configurations?
    On identical logged query trajectories, please compare query-time versus terminal predictions, each combined with either arithmetic averaging or LV aggregation, and report paired error differences. In particular, how much of the improvement remains when the terminal contributions are averaged directly, corresponding to c=0c=0? This would clarify the respective roles of refitting and aggregation.

  2. Where precisely does the theoretical advance lie relative to the closest existing results?
    A concise comparison with AISCV, original LV, and martingale-based truncation or Catoni methods would be helpful. Relevant dimensions include the permitted dependence structure, the fitted function class, the treatment of the preliminary estimate and calibration, and the resulting guarantee. Numerical examples of the bounds under the stated assumptions would also clarify their practical significance, without requiring them to be presented as observable confidence intervals.

  3. What separation between historical, development, and evaluation models can be verified?
    Are there results using fully disjoint model identities or a leave-one-model-family-out design? If so, do the improvements persist relative to a strong regression-assisted baseline and direct averaging of terminal contributions? Where identities cannot be established, please specify which conclusions remain restricted to the evaluated cohorts; distinct response vectors alone do not establish independence between models.

Reviewer_rzz7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes the Adaptive Surrogate-Augmented Lee-Valiant (ASALV) estimator for efficient benchmark prediction. The method leverages the fact that the score of every answer is bounded and that standard prediction residuals do not natively account for this.

The authors claim to have finite-sample error bounds for ASALV under adaptive querying and reuse of queried answers for surrogate fitting.

The authors apply their method to two datasets and show that it is better (under both MAE and RMSE) than the baselines chosen. They also evaluate the robustness of their method to label noise.

Critical strengths

The problem the paper addresses is extremely important. They are right: model and agent evaluations can cost thousands of dollars and so we should try to make it more resource efficient.

The idea of using properties of the scores to improve score estimation is sound.

Critical weaknesses

The writing is not easy to follow and assumes the reader has non-trivial knowledge about the problem. I found the first paragraph in particular to do a poor job of introducing the problem. Similarly, the second paragraph uses pretty non-standard terms without and references or definitions that make it hard to parse.

The theorem, while ambitious, is stated weirdly. For example, I do not know what "exact arithmetic" means in the context of measure theory or statistical pattern recognition. I would also expect the authors to engage with the theorem in more digestible prose after the statement to solidify our understanding.

Lastly, the empirical analysis is done on two datasets. Both of which are pretty outdated in the context of expensive model evaluations. I'd expect HELM, EEE, TerminalBench, SWEBench, etc. instead of MMLU. Along these lines, the others do not consider serious baselines (including basic IRT itself!) and so it is hard to assess the proposed methods place in the context of the literature.

I would also expect more analysis in the hyper efficient (<= 1%) and small number of models regimes.

Questions for authors

Given that the writing, technical material, and empirical results are all below the standard of ICLR I do not expect the authors to be able to change my scores meaningfully during a rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.