- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
ASA-LV estimates full-benchmark accuracy from few-answer queries. It reconstructs propensity-corrected contributions with a terminal surrogate and applies Lee–Valiant aggregation. The analysis allows adaptive querying and reuse of queried answers for surrogate fitting. Experiments cover 19 benchmarks and the FAQ data.
Critical strengths
-
Estimating full-benchmark accuracy with few-answer queries addresses an important practical problem in model evaluation. The estimator itself is simple. It reconstructs augmented contributions using the terminal surrogate and then applies LV aggregation.
-
The analysis allows adaptive querying and reuse of queried answers for surrogate fitting, and quantifies the effects of surrogate-class complexity and covering approximation. The evaluation covers 19 benchmarks and the FAQ data. ASA-LV reports lower RMSE and MAE than the compared estimators at both query budgets.
Critical weaknesses
-
The main argument is difficult to follow. The introduction places propensity-amplified residuals alongside the data dependence introduced by terminal fitting, without first explaining why terminal reconstruction is needed. The motivation for combining these components, and the contribution relative to FAQ, remain unclear.
-
The abstract gives an estimation-error rate without defining k. Section 3 then introduces many symbols and conditions through dense inline mathematics. The problem is simple, but its formulation places an unnecessary burden on the reader. Section 4 introduces dense technical details before making clear what the authors want to do and why. I found these explanations distracting because they obscured the main idea before I could grasp the intuition behind the method. Section 5 gives a proof outline and a short explanation of the nonlinear correction, but little guidance on the importance of the results. It does not provide a coherent interpretation of the error terms or clearly connect the two theorems to the experimental configuration. The derivations are difficult to assess as contributions.
Questions for authors
-
Why reconstruct all contributions with the terminal surrogate rather than retain FAQ’s query-time predictors? What benefit justifies the resulting data dependence?
-
Why use LV aggregation instead of simply averaging the augmented contributions? What benefit does it provide in this setting? Is it worth it? Is it so important that it should be the main contribution and appear directly in the paper's title?
-
What do the authors want readers to learn from the theoretical analysis? According to my understanding, the authors intend to enhance the conclusion that LV makes sense and bias caused using a terminal surrogate is controlled. Why not organize the sections more clearly?
-
Do the experiments separately establish the benefits of terminal reconstruction and LV aggregation over FAQ? How much does each change contribute to the overall improvement?