acceptodds
Under review as a conference paper at ICLR 2027

You Don't Need the Whole Meta-Dataset: Budgeted Model Evaluation on Unseen and Unlabeled Data

Abstract

Predicting model performance on an unseen and unlabeled workload commonly starts by measuring its distribution shift from the training data. To convert this shift into an accuracy estimate, prior works train an evaluator on a diverse meta-dataset, using the shift between each subset of the meta-dataset and the training data as input, and the model accuracy on that subset as the label. However, obtaining these labels is expensive because each model must be run and evaluated on every subset. The more subsets the meta-dataset contains, the more forward passes are required from each reference model to obtain its accuracy labels before the evaluator can be trained. To overcome this limitation, we present ActiveEvaluator, which trains the same evaluator on a small representative subset of the meta-dataset instead of the full one, reducing the cost and latency of model evaluation to nearly one third while matching or even outperforming full-set training. ActiveEvaluator measures subset similarity using Hausdorff distance and maximizes a facility-location objective that assigns each subset to its closest representative, giving redundant subsets little marginal gain. The objective is monotone and submodular, so greedy selection comes with a provable performance guarantee. We validate ActiveEvaluator across Text2SQL, image classification, and node classification, showing that the same budgeted selection principle generalizes across structurally different tasks. As data and model pools grow faster than the resources available to evaluate them, this opens a direction toward fast, budgeted evaluation of new models on unlabeled data. It also makes evaluation more practical in domains where data are limited. Our code is available at https://anonymous.4open.science/r/ActiveEvaluator-00B1/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
model evaluation, budget-constrained evaluation, submodular maximization, unlabeled data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ijFJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the cost of evaluating unseen models on unlabeled datasets using meta-learned performance estimators. The authors propose ActiveEvaluator, which selects representative workloads from a large meta-dataset before obtaining reference-model performance labels. It uses contextual embeddings, Hausdorff distance, and greedy facility-location maximization for budgeted selection. The theoretical analysis establishes submodularity and a greedy approximation guarantee, along with conditional bounds relating workload coverage to performance and gradient approximation. Experiments on Text2SQL, image classification, and node classification show that using approximately one-third of the meta-dataset largely preserves the accuracy-estimation and model-ranking performance of full MetaEvaluator.

Critical strengths
  1. The paper addresses the substantial computational cost of constructing supervision for meta-learned model evaluation. Its modular selection approach can be integrated with existing evaluators and potentially reduces expensive model-workload evaluations.
  2. Experiments across Text2SQL, image classification, and node classification demonstrate that reduced meta-datasets can retain performance-estimation and ranking quality. The consistent reduction in supervision requirements across different modalities is a useful practical finding.
Critical weaknesses
  1. The proposed approach combines Hausdorff distance with standard facility-location selection. The submodularity and greedy approximation guarantee follow established results. While the application to meta-evaluation is relevant, the algorithmic and theoretical novelty appears relatively limited.
  2. The experiments primarily compare ActiveEvaluator using fewer workloads against existing evaluators trained on the full meta-dataset. Comparisons with random sampling, clustering-based selection, k-center, and conventional facility location at identical budgets are missing. These are important for establishing the specific benefit of the proposed selection strategy.
  3. Computing pairwise Hausdorff distances and performing greedy selection over a 30K-workload pool may introduce substantial preprocessing overhead. The reported efficiency results lack a sufficiently detailed breakdown of embedding, distance computation, and selection costs, making the end-to-end savings difficult to assess.
  4. The main approximation guarantee applies to geometric coverage rather than the final estimation error. The additional bounds rely on smoothness assumptions connecting workload similarity to model performance and evaluator gradients, without demonstrating that these assumptions hold empirically. Thus, the theoretical results provide only indirect support for preserving evaluation accuracy.
Questions for authors
  1. How does ActiveEvaluator compare with random sampling, clustering-based methods, and standard facility-location selection under identical workload budgets?
  2. What are the end-to-end costs of workload embedding, pairwise distance computation, and greedy selection? Are these included in the reported latency improvements?
  3. Can the authors empirically validate the relationship between Hausdorff workload distance, model performance, and evaluator gradients?
  4. How sensitive is the selection strategy to embedding choices, noisy workloads, and outliers? Do the conclusions remain consistent across alternative workload representations?
Reviewer_KZ7FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ActiveEvaluator aims to cut the supervision needed to train meta-learned accuracy estimators for unseen models on unlabeled data. Rather than labelling every workload of a 30K meta-dataset with every reference model, it represents each workload as a set of frozen token embeddings, measures workload similarity with the symmetric Hausdorff distance, and greedily maximizes a facility-location objective to select KK representatives (with the standard 1−1e1 - \frac{1}{e} guarantee on coverage). MetaEvaluator is then trained on the selected workloads, weighted by how many workloads each represents. On Text2SQL (seven targets) and, in the appendix, image and node classification, selecting 8K–10K of 30K workloads gives an average MAE close to full-set training (5.59 vs. 5.51, 3.74 vs. 3.64 and 3.60 vs. 3.40) with similar ranking quality. The abstract claims that cost and latency drop to nearly one third while matching or even outperforming full-set training.

Critical strengths

1. A practical supervision-budget problem. Obtaining accuracy labels for meta-learned evaluators requires running reference models on workloads, so reducing this supervision is a worthwhile objective. The proposed selection step is simple, independent of reference-model predictions and reusable across reference models.

2. Evaluation across modalities with explicit trade-offs. The reported results cover three modalities, seven targets per modality, ten runs and several ranking metrics. The small average MAE gaps suggest that workload supervision may be compressible, and the paper reports targets where the subset performs worse, including Spider 2.0, BEAVER and ScienceBench. The submodularity proof (Appendix A) and the conditional gradient bound (Appendix B, Eqs. 17–20) are correct under their stated assumptions.

Critical weaknesses

1. The claimed end-to-end efficiency savings are not established.

  • Workloads are sampled from a shared labeled pool (Spider and BIRD in the released configuration), and the implementation caches model outputs per example. As a diagnostic, applying the released generator to the bundled Spider-only pool (8,659 examples; seed 0) produces 30K workloads with 3,349,178 example slots, or 386.8 occurrences per example on average. The first 1K generated workloads cover 99.49% of the pool, and the first 3K cover all of it. This substantial overlap shows why a reduction in workload count need not yield comparable savings in unique model–input evaluations. It does not establish the actual savings of the Hausdorff-selected 9K workloads on the complete Spider+BIRD pool; those require the selected workload IDs and deduplicated input counts.
  • The released timing path sums cached per-example times within each workload (text2sql/run.py, L103–107) and then across workloads (active_evaluator/protocol.py, L54–55), repeatedly counting shared inputs. Encoding and selection times are recorded separately but are not included in these experiment timing fields. These fields therefore do not establish end-to-end wall-clock savings. Fig. 6 itself shows approximately 80 vs. 118 minutes at 20 unseen models (ratio≈0.68\text{ratio} \approx 0.68), which does not support the abstract's nearly-one-third latency claim under that setting.

2. The selection rule's added value and the validity of the evaluation protocol remain unresolved.

  • There is no same-KK random or stratified selection control using the same MetaEvaluator and training budget. Table 2 compares meta-learning algorithms and workload distances, but does not isolate the benefit of the proposed selection rule over simpler sampling. The four Table 1 improvements over MetaEvaluator are only 0.04–0.05 percentage points. If the reported intervals use the released tt-based implementation on the same ten paired runs, the paired-difference interval's half-width is at least the absolute difference between the two marginal half-widths. The resulting lower bounds are 0.24–0.33 points, so each paired interval would include zero even under the most favourable correlation. The point estimates can be described as lower, but they do not establish a significant advantage under this statistical convention; the noise-removal explanation (pp. 2, 8) also lacks a controlled test.
  • The code contains target-label information paths that need reconciliation with p. 8's isolation statement. ACM, Citation and DBLP form a source–target cycle, so the shared full meta-dataset includes labeled subgraphs from these three evaluation graphs through other tasks (configs/node.json, L11–24; node/run.py, L41–47, L116–123). Exposure in the selected subset depends on its workload IDs. In Text2SQL, the gold query's ORDER BY flag affects the prediction hash supplied to agreement baselines (text2sql/execution.py, L62–69; text2sql/run.py, L115); this does not show that ActiveEvaluator's target descriptors directly use gold labels. Separately, the executor can return EX as EX-S when all variants are absent, without enforcing the five variants required by Section 5.1. These are concrete issues in the release, not evidence that the reported runs necessarily used the same paths or lacked variants.
Questions for authors
  1. Cost clarification: Per reference model, how many distinct input examples must be processed for the selected 9K and full 30K meta-datasets? Please report end-to-end wall-clock time with identical caching and batching, accounting for encoding, distances, selection, model inference, SQL execution, evaluator training, adaptation and target evaluation. Distinguish one-time from amortized costs. If the main savings concern evaluator training or adaptation rather than reference-model inference, please narrow the efficiency claim accordingly.
  2. Decisive comparison: Can you compare ActiveEvaluator with repeated same-KK random or stratified selection using the same MetaEvaluator, splits and training steps? Please report paired differences between the selection strategies, as well as their gaps to full-set training.
  3. Results and protocol clarification: Please map Fig. 1 (3.66/3.82), Table 1 (5.51/5.59), Table 2's left panel (4.35/4.46) and right panel (Hausdorff: 3.47), and Fig. 6 to their datasets, model splits, budgets and seeds. How did the reported runs prevent the target-label dependencies described above and enforce all five EX-S variants? Please also identify the actual model/reference pools behind the 20-model node and image results: the released node configuration contains only 16 architectures, and selecting all 20 image models as unseen leaves no reference models under the released split procedure.
Reviewer_DFxLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ActiveEvaluator selects representative workloads from a historical pool before running reference models to collect accuracy labels. It uses Hausdorff distances and a greedy facility-location objective to choose the workloads, then trains an existing performance evaluator on the weighted subset. Across Text-to-SQL, image classification, and node classification, the reported mean prediction errors remain close to those obtained with the full pool.

Critical strengths
  1. The paper poses a fresh and practical question: when collecting model scores is expensive, which historical workloads are worth evaluating to train a reliable performance predictor?
  2. The paper is well organized and easy to follow. Figure 3 clearly shows how workload selection fits into evaluator training, and Algorithm 1 makes the selection procedure easy to trace.
  3. The evaluation covers three tasks with seven target datasets each. The authors assess both accuracy estimation and model ranking, which matters when the goal is to choose among candidate models.
Critical weaknesses
  1. The experiments do not isolate the benefit of the selection rule. The main methodological contribution is applying an established facility-location objective to whole workloads before collecting model scores (§4.3). Table 2 compares distance functions, but the paper reports no comparison against random sampling or clustering at the same 9K budget. It therefore remains unclear how much the result depends on which workloads are selected.
  2. The end-to-end savings are difficult to assess. For 30K workloads, Algorithm 1 specifies roughly 450 million pairwise distances before greedy selection begins. Appendix B counts the training work saved after selection, while Figure 6 does not specify whether its timing includes embedding the workloads, computing distances, and selecting representatives.
  3. The evidence for transfer to an entirely new data source is limited by how the splits are described. Section 5.1 states that target labels are hidden, but does not specify whether the historical Text-to-SQL workloads and test targets are disjoint by database schema or source. That distinction matters when interpreting performance on unseen workloads.
Questions for authors
  1. With the evaluator and training protocol fixed, how does MAE at 9K workloads compare across the proposed selector, uniform random sampling over several seeds, and clustering?
  2. Which preprocessing stages are included in Figure 6? How long does selection take, and after how many evaluated models does the cumulative time saving offset that cost?
  3. Which Text-to-SQL schemas or data sources are held out entirely from the historical pool? On those targets, how closely does the 9K subset match the full 30K pool?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.