- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CollabEval uses historical model evaluations to reduce the number of prompts needed to evaluate new models. It completes a partially observed model-by-prompt score matrix and uses the predictions as control variates with bias correction. It estimates both model scores and differences between models. Tests on five benchmarks show narrower confidence intervals and lower estimation error at the same evaluation budget.
Critical strengths
Historical evaluations can reduce both generation and scoring costs for new models. Using imputed scores as control variates avoids treating potentially biased predictions as ground truth. The experiments across five benchmarks show that this gives useful reductions in estimation error and confidence-interval width at the same evaluation budget.
Critical weaknesses
The PPI baseline using an untrained average of anchor scores is not strong enough to establish the need for matrix completion. A learned, target-specific regression on anchor scores, using the same cross-fitting and bias correction, would be a more informative comparison. The results demonstrate the value of historical information more clearly than the added value of the proposed matrix-completion approach.
Questions for authors
What does matrix completion add over a learned, target-specific regression on anchor scores, using the same cross-fitting, bias correction, and power tuning? Is this advantage established theoretically or isolated in any reported comparison?
Author response
Thank you for the review and the positive assessment! We’d like to clear up some of your questions and concerns to hopefully help raise your confidence in our work.
With respect to your main question: what does matrix completion add over a target-specific regression on anchor scores? Our framework is not tied to a particular matrix completion algorithm. Any stable method (see Assumption A.1) that predicts the missing scores can be plugged in, and cross-fitting, bias correction, power tuning, and our handling of overlapping samples for pairwise comparisons keep inference asymptotically valid regardless. A target-specific regression on anchor scores (that is appropriately regularized) is one such method (the PPI baseline with anchor score is meant as an example of a baseline that just uses a control variate that is already provided without any fitting).
Appendix D.2 already compares IterativeSVD to a neural network, nuclear norm minimization, and a 2-PL IRT model. We use IterativeSVD because it is simple and robust, and we do not claim it is the best choice. Following your suggestion, we also ran CollabEval with a per-target ridge regression as the completion method (CollabEval-Ridge), with everything else unchanged (now in Appendix D.3). It performs about the same as IterativeSVD, provided that it is properly regularized.
Average 5 task performance. Individual model scores (θ), % MSE reduction relative to Classic (IID), with CI coverage in parentheses:
Additionally, note that matrix completion falls more generally under prediction with missing features, which IterativeSVD handles quite naturally. We have added an experiment to the appendix (Appendix C.6) that hides a random fraction of the anchor entries, and show that coverage is maintained at every sparsity level while efficiency degrades gracefully. Ridge and similar regression methods would require choosing how to impute the missing features separately (though this is of course possible -- even, for example, by first imputing with iSVD, and then running Ridge).
Average 5 task performance of CollabEval (iSVD, from a separate set of runs) when a random fraction of the anchor entries is hidden (columns). Individual model scores (θ), % MSE reduction relative to Classic (IID), with CI coverage in parentheses: