- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the cost of evaluating unseen models on unlabeled datasets using meta-learned performance estimators. The authors propose ActiveEvaluator, which selects representative workloads from a large meta-dataset before obtaining reference-model performance labels. It uses contextual embeddings, Hausdorff distance, and greedy facility-location maximization for budgeted selection. The theoretical analysis establishes submodularity and a greedy approximation guarantee, along with conditional bounds relating workload coverage to performance and gradient approximation. Experiments on Text2SQL, image classification, and node classification show that using approximately one-third of the meta-dataset largely preserves the accuracy-estimation and model-ranking performance of full MetaEvaluator.
Critical strengths
- The paper addresses the substantial computational cost of constructing supervision for meta-learned model evaluation. Its modular selection approach can be integrated with existing evaluators and potentially reduces expensive model-workload evaluations.
- Experiments across Text2SQL, image classification, and node classification demonstrate that reduced meta-datasets can retain performance-estimation and ranking quality. The consistent reduction in supervision requirements across different modalities is a useful practical finding.
Critical weaknesses
- The proposed approach combines Hausdorff distance with standard facility-location selection. The submodularity and greedy approximation guarantee follow established results. While the application to meta-evaluation is relevant, the algorithmic and theoretical novelty appears relatively limited.
- The experiments primarily compare ActiveEvaluator using fewer workloads against existing evaluators trained on the full meta-dataset. Comparisons with random sampling, clustering-based selection, k-center, and conventional facility location at identical budgets are missing. These are important for establishing the specific benefit of the proposed selection strategy.
- Computing pairwise Hausdorff distances and performing greedy selection over a 30K-workload pool may introduce substantial preprocessing overhead. The reported efficiency results lack a sufficiently detailed breakdown of embedding, distance computation, and selection costs, making the end-to-end savings difficult to assess.
- The main approximation guarantee applies to geometric coverage rather than the final estimation error. The additional bounds rely on smoothness assumptions connecting workload similarity to model performance and evaluator gradients, without demonstrating that these assumptions hold empirically. Thus, the theoretical results provide only indirect support for preserving evaluation accuracy.
Questions for authors
- How does ActiveEvaluator compare with random sampling, clustering-based methods, and standard facility-location selection under identical workload budgets?
- What are the end-to-end costs of workload embedding, pairwise distance computation, and greedy selection? Are these included in the reported latency improvements?
- Can the authors empirically validate the relationship between Hausdorff workload distance, model performance, and evaluator gradients?
- How sensitive is the selection strategy to embedding choices, noisy workloads, and outliers? Do the conclusions remain consistent across alternative workload representations?