- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PoolEvaluator, a framework for estimating and ranking the accuracy of multiple black-box models on an unlabeled target dataset. It uses cross-model agreement, optionally supplemented by labeled calibration datasets and external judge verdicts, to reduce reliance on target ground-truth labels. An EM procedure estimates model accuracies and agreement rates, while selective judging refines the estimates.
Critical strengths
- Practical problem. Estimating model performance and selecting among candidate models when target labels are scarce or expensive is an important evaluation problem.
- Potentially useful evaluation signal. Cross-model agreement provides information beyond evaluating each model independently. Combining this signal with calibration data and selective judging is a promising approach.
- Broad empirical coverage. The experiments span Text2SQL, image classification, and node classification across multiple datasets, measuring both accuracy estimation and ranking quality.
Critical weaknesses
- Identifiability of accuracy from agreement.
My primary concern is whether the stated model can identify accuracy from unlabeled agreement observations. Marginalizing Equation 1 gives
This constrains one probability while three parameters are unknown. For example, both and imply an agreement probability of 0.7 despite different accuracies. Both also satisfy .
Under the factorized likelihood in Equation 2, I do not see an additional constraint that resolves this ambiguity. Calibration-based initialization may influence which solution EM reaches, but initialization alone does not establish identifiability. This is particularly relevant when a strong model disagrees with a weaker majority or several models share systematic errors.
-
The contribution of unlabeled agreement needs clearer separation from labeled information.
The default configuration uses labeled calibration subsets and judge-provided correctness assignments. The paper also permits random initialization and claims competitive performance with both budgets set to zero (, Section 6).Figure 7 varies the budgets separately, but explicit results for the joint zero-budget setting would help assess this claim. A comparison of calibration-only estimates, calibration plus EM, and calibration plus EM plus judging would also clarify how much the unlabeled agreement stage contributes. The distinction between evaluation on an unlabeled target and evaluation without any labeled information should be explicit.
-
Judge verdicts are treated as exact correctness information.
Equations 3 and 9 assign hard correctness labels from the external judge, while Appendix A assumes exact validation. In practice, judges may make mistakes or share errors with the evaluated models. Using a different model family does not by itself establish independent errors.Reporting judge accuracy on the selected examples and assessing sensitivity to erroneous verdicts would strengthen the evaluation. Practical guidance on validating judge quality and handling disagreement between judges would also improve the deployment case.
Questions for authors
- What assumptions or constraints identify separately from and ? How does the method distinguish parameter settings with identical agreement probabilities but different accuracies?
- Can you provide explicit results for and isolate the improvement from EM over calibration-only estimates?
- How accurate are the judges on the queried examples, and how sensitive are accuracy estimates and rankings to judge errors? How should practitioners evaluate this risk on a new dataset?
Thank you for addressing most of my concerns. However, I have a follow-up question regarding the following:
You have compared PoolEvaluator against Model Selector on Text2SQL, whereas Model Selector is designed for classification tasks. How did you adapt Model Selector to these non-classification tasks? In my opinion, it would be more appropriate to evaluate on tasks for which the method was originally designed, such as image classification.